Python course Β· Module 12: Final Project

AI Testing Strategy

6 min read Β·
In this lesson5

A ranger who only checks that the Land Rover starts in the car park still does not know whether it will cross the river. Testing AI applications has the same problem in a double dose. Ordinary code behaves deterministically: the same function with the same input gives the same result. A language model can answer the same question in two different ways. That is why we have to verify not only the code, but also the quality of the answers.

The Test Pyramid for AI

The classic test pyramid says: most of your tests should be fast unit tests at the base, fewer integration tests in the middle, and the fewest slow end-to-end tests at the top. In an AI project we add a layer that evaluates the model's answers. The proportions below are a rough target, not a rule:

                    β•±β•²
                   β•±  β•²
                  β•± E2Eβ•²           ← 10% - End-to-end tests
                 ╱──────╲
                β•±        β•²
               β•±Integrationβ•²       ← 20% - Integration tests
              ╱────────────╲
             β•±              β•²
            β•± AI Evaluation  β•²     ← 30% - AI evaluation
           ╱──────────────────╲
          β•±                    β•²
         β•±        Unit          β•²  ← 40% - Unit tests
        ╱────────────────────────╲

The base is the widest because unit tests are the fastest and cheapest. AI evaluation sits above them: it is slower and often costs money (every question is a model call). From cheapest to most expensive: unit, integration, E2E, and finally performance tests under load.

Unit Tests

We write unit tests in pytest. A fixture is a function marked with @pytest.fixture that prepares data or resources for a test, and pytest injects it by parameter name. Mock from unittest.mock creates a fake object, and spec= makes sure the fake only has the methods of the real class.

import pytest
from unittest.mock import Mock, AsyncMock

class TestDocumentService:
    @pytest.fixture
    def mock_repo(self):
        return Mock(spec=DocumentRepository)

    @pytest.fixture
    def mock_embedder(self):
        embedder = Mock()
        embedder.embed.return_value = [0.1] * 1536
        return embedder

    @pytest.fixture
    def service(self, mock_repo, mock_embedder):
        return DocumentService(mock_repo, mock_embedder)

The service fixture itself uses two other fixtures, so pytest builds the whole dependency tree for you. The embedder returns a fixed vector of 1536 numbers without any API call. The rest of the same class holds the actual tests:

    def test_chunk_document(self, service):
        doc = Document(content="A" * 1000)
        chunks = service.chunk_document(doc, chunk_size=200)

        assert len(chunks) == 5
        assert all(len(c.content) <= 200 for c in chunks)

    @pytest.mark.asyncio
    async def test_save_document(self, service, mock_repo):
        doc = Document(title="Test", content="Content")
        mock_repo.save = AsyncMock(return_value=doc)

        result = await service.save(doc)

        mock_repo.save.assert_called_once()
        assert result.title == "Test"

The chunking test checks pure logic: 1000 characters in pieces of 200 gives 5 chunks. The save test is asynchronous, so it carries the @pytest.mark.asyncio marker from the pytest-asyncio plugin and uses AsyncMock, which returns a value after await. An assertion is simply assert result == expected or, as here, assert result.title == "Test". To replace a function in a module (for example an external API call) you use unittest.mock.patch, either as a decorator or as with patch("module.function") as fake:.

Integration Tests

An integration test runs several parts together. Testcontainers starts a real Qdrant in a Docker container for the duration of the tests, and httpx.AsyncClient sends requests straight to the ASGI application, without a network.

import pytest
import pytest_asyncio
from httpx import ASGITransport, AsyncClient
from testcontainers.community.qdrant import QdrantContainer

@pytest.fixture(scope="module")
def qdrant_container():
    with QdrantContainer() as qdrant:
        yield qdrant

@pytest_asyncio.fixture
async def app_client(qdrant_container):
    app = create_app(qdrant_url=f"http://{qdrant_container.rest_host_address}")
    transport = ASGITransport(app=app)
    async with AsyncClient(transport=transport, base_url="http://test") as client:
        yield client

Two things have changed in the libraries and are worth knowing. Since httpx 0.28 there is no app= parameter anymore, you pass the application through ASGITransport. In pytest-asyncio, an asynchronous fixture is marked with @pytest_asyncio.fixture. You install the Qdrant container as testcontainers[qdrant], and older versions imported it from testcontainers.qdrant. Now the test itself:

@pytest.mark.asyncio
async def test_upload_and_query(app_client):
    # Upload document
    response = await app_client.post(
        "/documents",
        json={"title": "Test", "content": "Python is great"}
    )
    assert response.status_code == 201

    # Query
    response = await app_client.post(
        "/query",
        json={"question": "What is Python?"}
    )
    assert response.status_code == 200
    assert "Python" in response.json()["answer"]

The test walks through the whole flow: upload, then query. Notice that the last assertion depends on the model's answer, so it can be flaky. Checks like that are better moved to evaluation.

AI Evaluation

Evaluation measures answer quality on a prepared set of questions. Each case contains a question and the keywords we expect in the answer:

from dataclasses import dataclass

@dataclass
class EvalCase:
    question: str
    expected_keywords: list[str]
    context_required: bool = True

eval_dataset = [
    EvalCase(
        question="What is RAG?",
        expected_keywords=["retrieval", "generation", "augmented"],
        context_required=True
    ),
    EvalCase(
        question="How does embedding work?",
        expected_keywords=["vector", "semantic", "representation"],
        context_required=True
    )
]

The dataset itself is plain data. The scoring function computes what share of the keywords appeared in the answer and checks whether the system returned sources:

async def evaluate_rag_system(query_fn, dataset: list[EvalCase]) -> dict:
    results = {"total": len(dataset), "passed": 0, "failed": []}

    for case in dataset:
        response = await query_fn(case.question)

        # Check keywords
        answer_lower = response.answer.lower()
        keywords_found = sum(1 for k in case.expected_keywords if k in answer_lower)
        keyword_score = keywords_found / len(case.expected_keywords)

        # Check sources
        has_sources = len(response.sources) > 0 if case.context_required else True

        if keyword_score >= 0.5 and has_sources:
            results["passed"] += 1
        else:
            results["failed"].append({
                "question": case.question,
                "keyword_score": keyword_score,
                "has_sources": has_sources
            })

    results["accuracy"] = results["passed"] / results["total"]
    return results

The result is an accuracy value plus a list of failures to review. Keywords are a simple, rough method. More serious projects also score how faithful the answer is to its sources, but the principle stays: a fixed set of questions, a repeatable measurement.

CI/CD Pipeline

Tests make sense when they run on their own on every push. GitHub Actions runs the steps on a clean machine, with Qdrant running alongside as a service:

# .github/workflows/test.yml
name: Test Suite

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    services:
      qdrant:
        image: qdrant/qdrant
        ports:
          - 6333:6333

    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Install dependencies
        run: pip install -r requirements.txt

      - name: Run unit tests
        run: pytest tests/unit -v

      - name: Run integration tests
        run: pytest tests/integration -v
        env:
          QDRANT_URL: http://localhost:6333

      - name: Run AI evaluation
        run: python scripts/evaluate.py
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

The API key comes from secrets, never from the repository. Code coverage is measured by the pytest-cov plugin: pytest --cov=app lists which lines were never tested. You run a single file with pytest tests/unit/test_service.py, and many pipelines also build an image after the tests with docker build -t myapp:latest ., which we will come back to in the deployment lesson. My advice: run the AI evaluation separately from the unit tests, because it is slower and costs money. In the next lesson we will work on documentation.

Remember: tests are trial crossings of the river before you lead the whole expedition through it.

Docs entries used in this lesson

Spotted a mistake in this lesson?

Check yourself

Answer the questions from this lesson. Pick an answer to see right away whether it is correct.

  1. 1. What is at the base of the test pyramid?

  2. 2. What are pytest fixtures used for?

These are 2 of 3 questions for this lesson. Solve the rest in the game.

Hands-on tasks in the game

  • Vertical ordering

    Arrange the types of tests from fastest to slowest:

  • Code editor

    Write a unit test for a data processing function using pytest

  • Click in order

    Click the elements to build a pytest assert statement:

  • Horizontal ordering

    Arrange the Docker build command in the correct order:

  • Code editor

    Create a test using Mock to simulate an external API dependency

  • Vertical ordering

    Arrange the AI test pyramid layers from the base to the top:

  1. QuestionTest pyramid
  2. QuestionPytest fixtures
  3. Put in orderTypes of tests
  4. Coding taskUnit test with pytest
  5. QuestionPytest run
  6. Click in orderPytest assert
  7. Put in orderDocker build
  8. Coding taskMock in pytest
  9. Put in orderAI test pyramid

Useful articles