Python course Β· Module 12: Final Project
AI Testing Strategy
In this lesson5
A ranger who only checks that the Land Rover starts in the car park still does not know whether it will cross the river. Testing AI applications has the same problem in a double dose. Ordinary code behaves deterministically: the same function with the same input gives the same result. A language model can answer the same question in two different ways. That is why we have to verify not only the code, but also the quality of the answers.
The Test Pyramid for AI
The classic test pyramid says: most of your tests should be fast unit tests at the base, fewer integration tests in the middle, and the fewest slow end-to-end tests at the top. In an AI project we add a layer that evaluates the model's answers. The proportions below are a rough target, not a rule:
β±β²
β± β²
β± E2Eβ² β 10% - End-to-end tests
β±βββββββ²
β± β²
β±Integrationβ² β 20% - Integration tests
β±βββββββββββββ²
β± β²
β± AI Evaluation β² β 30% - AI evaluation
β±βββββββββββββββββββ²
β± β²
β± Unit β² β 40% - Unit tests
β±βββββββββββββββββββββββββ²The base is the widest because unit tests are the fastest and cheapest. AI evaluation sits above them: it is slower and often costs money (every question is a model call). From cheapest to most expensive: unit, integration, E2E, and finally performance tests under load.
Unit Tests
We write unit tests in pytest. A fixture is a function marked with @pytest.fixture that prepares data or resources for a test, and pytest injects it by parameter name. Mock from unittest.mock creates a fake object, and spec= makes sure the fake only has the methods of the real class.
import pytest
from unittest.mock import Mock, AsyncMock
class TestDocumentService:
@pytest.fixture
def mock_repo(self):
return Mock(spec=DocumentRepository)
@pytest.fixture
def mock_embedder(self):
embedder = Mock()
embedder.embed.return_value = [0.1] * 1536
return embedder
@pytest.fixture
def service(self, mock_repo, mock_embedder):
return DocumentService(mock_repo, mock_embedder)The service fixture itself uses two other fixtures, so pytest builds the whole dependency tree for you. The embedder returns a fixed vector of 1536 numbers without any API call. The rest of the same class holds the actual tests:
def test_chunk_document(self, service):
doc = Document(content="A" * 1000)
chunks = service.chunk_document(doc, chunk_size=200)
assert len(chunks) == 5
assert all(len(c.content) <= 200 for c in chunks)
@pytest.mark.asyncio
async def test_save_document(self, service, mock_repo):
doc = Document(title="Test", content="Content")
mock_repo.save = AsyncMock(return_value=doc)
result = await service.save(doc)
mock_repo.save.assert_called_once()
assert result.title == "Test"The chunking test checks pure logic: 1000 characters in pieces of 200 gives 5 chunks. The save test is asynchronous, so it carries the @pytest.mark.asyncio marker from the pytest-asyncio plugin and uses AsyncMock, which returns a value after await. An assertion is simply assert result == expected or, as here, assert result.title == "Test". To replace a function in a module (for example an external API call) you use unittest.mock.patch, either as a decorator or as with patch("module.function") as fake:.
Integration Tests
An integration test runs several parts together. Testcontainers starts a real Qdrant in a Docker container for the duration of the tests, and httpx.AsyncClient sends requests straight to the ASGI application, without a network.
import pytest
import pytest_asyncio
from httpx import ASGITransport, AsyncClient
from testcontainers.community.qdrant import QdrantContainer
@pytest.fixture(scope="module")
def qdrant_container():
with QdrantContainer() as qdrant:
yield qdrant
@pytest_asyncio.fixture
async def app_client(qdrant_container):
app = create_app(qdrant_url=f"http://{qdrant_container.rest_host_address}")
transport = ASGITransport(app=app)
async with AsyncClient(transport=transport, base_url="http://test") as client:
yield clientTwo things have changed in the libraries and are worth knowing. Since httpx 0.28 there is no app= parameter anymore, you pass the application through ASGITransport. In pytest-asyncio, an asynchronous fixture is marked with @pytest_asyncio.fixture. You install the Qdrant container as testcontainers[qdrant], and older versions imported it from testcontainers.qdrant. Now the test itself:
@pytest.mark.asyncio
async def test_upload_and_query(app_client):
# Upload document
response = await app_client.post(
"/documents",
json={"title": "Test", "content": "Python is great"}
)
assert response.status_code == 201
# Query
response = await app_client.post(
"/query",
json={"question": "What is Python?"}
)
assert response.status_code == 200
assert "Python" in response.json()["answer"]The test walks through the whole flow: upload, then query. Notice that the last assertion depends on the model's answer, so it can be flaky. Checks like that are better moved to evaluation.
AI Evaluation
Evaluation measures answer quality on a prepared set of questions. Each case contains a question and the keywords we expect in the answer:
from dataclasses import dataclass
@dataclass
class EvalCase:
question: str
expected_keywords: list[str]
context_required: bool = True
eval_dataset = [
EvalCase(
question="What is RAG?",
expected_keywords=["retrieval", "generation", "augmented"],
context_required=True
),
EvalCase(
question="How does embedding work?",
expected_keywords=["vector", "semantic", "representation"],
context_required=True
)
]The dataset itself is plain data. The scoring function computes what share of the keywords appeared in the answer and checks whether the system returned sources:
async def evaluate_rag_system(query_fn, dataset: list[EvalCase]) -> dict:
results = {"total": len(dataset), "passed": 0, "failed": []}
for case in dataset:
response = await query_fn(case.question)
# Check keywords
answer_lower = response.answer.lower()
keywords_found = sum(1 for k in case.expected_keywords if k in answer_lower)
keyword_score = keywords_found / len(case.expected_keywords)
# Check sources
has_sources = len(response.sources) > 0 if case.context_required else True
if keyword_score >= 0.5 and has_sources:
results["passed"] += 1
else:
results["failed"].append({
"question": case.question,
"keyword_score": keyword_score,
"has_sources": has_sources
})
results["accuracy"] = results["passed"] / results["total"]
return resultsThe result is an accuracy value plus a list of failures to review. Keywords are a simple, rough method. More serious projects also score how faithful the answer is to its sources, but the principle stays: a fixed set of questions, a repeatable measurement.
CI/CD Pipeline
Tests make sense when they run on their own on every push. GitHub Actions runs the steps on a clean machine, with Qdrant running alongside as a service:
# .github/workflows/test.yml
name: Test Suite
on: [push, pull_request]
jobs:
test:
runs-on: ubuntu-latest
services:
qdrant:
image: qdrant/qdrant
ports:
- 6333:6333
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- name: Install dependencies
run: pip install -r requirements.txt
- name: Run unit tests
run: pytest tests/unit -v
- name: Run integration tests
run: pytest tests/integration -v
env:
QDRANT_URL: http://localhost:6333
- name: Run AI evaluation
run: python scripts/evaluate.py
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}The API key comes from secrets, never from the repository. Code coverage is measured by the pytest-cov plugin: pytest --cov=app lists which lines were never tested. You run a single file with pytest tests/unit/test_service.py, and many pipelines also build an image after the tests with docker build -t myapp:latest ., which we will come back to in the deployment lesson. My advice: run the AI evaluation separately from the unit tests, because it is slower and costs money. In the next lesson we will work on documentation.
Remember: tests are trial crossings of the river before you lead the whole expedition through it.
Docs entries used in this lesson
Spotted a mistake in this lesson?
Check yourself
Answer the questions from this lesson. Pick an answer to see right away whether it is correct.
1. What is at the base of the test pyramid?
2. What are pytest fixtures used for?
These are 2 of 3 questions for this lesson. Solve the rest in the game.
Hands-on tasks in the game
- Vertical ordering
Arrange the types of tests from fastest to slowest:
- Code editor
Write a unit test for a data processing function using pytest
- Click in order
Click the elements to build a pytest assert statement:
- Horizontal ordering
Arrange the Docker build command in the correct order:
- Code editor
Create a test using Mock to simulate an external API dependency
- Vertical ordering
Arrange the AI test pyramid layers from the base to the top: