Python course Β· Module 11: RAG and Multi-Agent Systems
Production RAG - Enterprise Systems
In this lesson8
Your RAG prototype works great on your laptop. Then a hundred people use it at once, the API bill grows by the hour, one question waits ten seconds, and when something breaks nobody knows at which stage. A camp for one night is not the same as a permanent expedition base.
Building production RAG systems requires taking four things into account at once: performance, scalability, monitoring and security. It is like designing an entire ecosystem - every element must work in harmony with the others.
Production RAG Architecture
The diagram shows the layers every request passes through:
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Production RAG Architecture β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β β
β ββββββββββββ ββββββββββββββββ ββββββββββββββββ β
β β Client βββββΆβ Load BalancerβββββΆβ API Gateway β β
β ββββββββββββ ββββββββββββββββ ββββββββββββββββ β
β β β
β βΌ β
β ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β RAG Service Layer β β
β β ββββββββββββββ ββββββββββββββ ββββββββββββββ β β
β β β Retrieval β β Augment β β Generation β β β
β β β Service β β Service β β Service β β β
β β βββββββ¬βββββββ βββββββ¬βββββββ βββββββ¬βββββββ β β
β ββββββββββΌββββββββββββββββΌββββββββββββββββΌβββββββββββββββββββ β
β β β β β
β ββββββββββΌββββββββ ββββββΌβββββ ββββββββΌβββββββ β
β β Vector Databaseβ β Cache β β LLM APIs β β
β β (Pinecone/ β β (Redis) β β (OpenAI/ β β
β β Qdrant) β β β β Anthropic) β β
β ββββββββββββββββββ βββββββββββ βββββββββββββββ β
β β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β Observability Layer β β
β β Prometheus β Grafana β Jaeger β LangSmith β Weights & Biases β β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββFrom the top: load balancer and API Gateway, the RAG service layer with its three stages, below that the vector database, the Redis cache and the model APIs, and under everything the observability layer.
FastAPI RAG Service
We write the service in FastAPI, because it supports async and validates data by itself. First the imports and the Pydantic models describing the request and the response:
from fastapi import FastAPI, HTTPException, BackgroundTasks
from pydantic import BaseModel
from typing import Optional
import asyncio
import hashlib
from contextlib import asynccontextmanager
import uvicorn
# Models
class QueryRequest(BaseModel):
question: str
top_k: int = 5
filters: Optional[dict] = None
class QueryResponse(BaseModel):
answer: str
sources: list[dict]
latency_ms: floatQueryRequest accepts a question, the number of documents and optional filters. FastAPI rejects a request without the question field by itself.
The service class connects to three services. The clients are asynchronous, so waiting for one service does not block other requests:
class RAGService:
"""Production RAG service."""
def __init__(self):
self.vector_store = None
self.llm = None
self.cache = None
async def initialize(self):
"""Initialize connections."""
# Vector store
from qdrant_client import AsyncQdrantClient
self.vector_store = AsyncQdrantClient(host="qdrant", port=6333)
# Cache
import redis.asyncio as redis
self.cache = redis.Redis(host="redis", port=6379)
# LLM
from openai import AsyncOpenAI
self.llm = AsyncOpenAI()We create the connections in initialize, because they need await. The host names qdrant and redis are the service names from the Docker Compose file you will see at the end of the lesson.
The heart of the service is the query method. It checks the cache first, and only on a miss runs the full pipeline:
async def query(self, request: QueryRequest) -> QueryResponse:
"""Processes a RAG query."""
import time
start = time.time()
# 1. Check cache
cache_key = f"rag:{hashlib.sha256(request.question.encode()).hexdigest()}"
cached = await self.cache.get(cache_key)
if cached:
return QueryResponse.model_validate_json(cached)
# 2. Embedding
embedding = await self._get_embedding(request.question)
# 3. Retrieval
docs = await self._retrieve(embedding, request.top_k, request.filters)
# 4. Generation
answer = await self._generate(request.question, docs)
# 5. Response
response = QueryResponse(
answer=answer,
sources=[{"text": d.payload["text"], "score": d.score} for d in docs],
latency_ms=(time.time() - start) * 1000
)
# 6. Cache
await self.cache.setex(cache_key, 3600, response.model_dump_json())
return responseWe compute the cache key with SHA-256, not the built-in hash(). This fix matters in production: hash() for strings is randomized at every process start, so two workers would compute different keys for the same question. setex stores the answer for 3600 seconds, one hour.
Three helper methods handle embedding, retrieval and generation:
async def _get_embedding(self, text: str) -> list[float]:
"""Generates an embedding."""
response = await self.llm.embeddings.create(
model="text-embedding-3-small",
input=text
)
return response.data[0].embedding
async def _retrieve(self, embedding, top_k, filters):
"""Retrieves documents."""
response = await self.vector_store.query_points(
collection_name="documents",
query=embedding,
limit=top_k,
query_filter=filters
)
return response.points
async def _generate(self, question: str, docs) -> str:
"""Generates an answer."""
context = "\n".join([d.payload["text"] for d in docs])
response = await self.llm.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": f"Context:\n{context}"},
{"role": "user", "content": question}
]
)
return response.choices[0].message.contentRetrieval uses query_points, because the search method is gone from new versions of qdrant-client. The document text lives in the payload, which is why we build sources from d.payload["text"].
Finally the FastAPI app and the endpoint:
# FastAPI app
rag_service = RAGService()
@asynccontextmanager
async def lifespan(app: FastAPI):
await rag_service.initialize()
yield
app = FastAPI(title="RAG API", lifespan=lifespan)
@app.post("/query", response_model=QueryResponse)
async def query(request: QueryRequest):
try:
return await rag_service.query(request)
except Exception as e:
raise HTTPException(status_code=500, detail=str(e))lifespan runs initialize once, at application start. Every error becomes a 500 code, but in production do not send str(e) back to the client.
Monitoring and Observability
Without measurements you learn about outages from your users. Prometheus collects three types of metrics: a counter, a histogram and a gauge:
from prometheus_client import Counter, Histogram, Gauge
import logging
import structlog
# Prometheus metrics
QUERY_COUNT = Counter(
"rag_queries_total",
"Total number of RAG queries",
["status"]
)
QUERY_LATENCY = Histogram(
"rag_query_latency_seconds",
"Query latency in seconds",
buckets=[0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0]
)
RETRIEVAL_SCORE = Gauge(
"rag_retrieval_score",
"Average retrieval score"
)
# Structured logging
structlog.configure(
processors=[
structlog.stdlib.add_log_level,
structlog.processors.TimeStamper(fmt="iso"),
structlog.processors.JSONRenderer()
]
)
logger = structlog.get_logger()A Counter only goes up, and the status label separates successes from errors. A Histogram sorts response times into buckets, and structlog writes logs as JSON.
The monitored service wraps processing in a timer:
class MonitoredRAGService:
"""RAG with monitoring."""
async def query(self, request: QueryRequest) -> QueryResponse:
with QUERY_LATENCY.time():
try:
response = await self._process_query(request)
QUERY_COUNT.labels(status="success").inc()
# Log
logger.info(
"query_completed",
question=request.question[:50],
latency_ms=response.latency_ms,
num_sources=len(response.sources)
)
return response
except Exception as e:
QUERY_COUNT.labels(status="error").inc()
logger.error("query_failed", error=str(e))
raiseQUERY_LATENCY.time() measures the duration of the with block, and an exception increments the error counter and propagates thanks to raise. _process_query stands for the logic of the service's query method.
LangChain Chains
In the production ecosystem you will often meet LangChain. Its basic pattern is a chain: a prompt connected to a model with the | operator and called with the invoke method:
from langchain_core.prompts import ChatPromptTemplate
from langchain_openai import ChatOpenAI
prompt = ChatPromptTemplate.from_template("Answer the question briefly: {input}")
chain = prompt | ChatOpenAI(model="gpt-4o-mini")
user_query = "What is a cache in a RAG system?"
result = chain.invoke({"input": user_query})
print(result.content)invoke takes a dictionary whose keys match the variables in the template, here {input}. The result is a model message, and the text is in the content field.
LangSmith Tracing
LangSmith is the LangChain team's platform for tracing calls. The @traceable decorator records the whole function, and trace creates nested stages inside it:
from langsmith import traceable, trace
from langsmith.wrappers import wrap_openai
from openai import OpenAI
# Every call through this client reaches LangSmith as a separate stage
openai_client = wrap_openai(OpenAI())
@traceable(name="RAG Query")
async def traced_query(question: str) -> str:
"""RAG query with tracing."""
# Embedding
with trace("embedding", metadata={"model": "text-embedding-3-small"}):
embedding = get_embedding(question)
# Retrieval
with trace("retrieval") as span:
docs = retrieve(embedding)
span.add_metadata({"num_docs": len(docs)})
# Generation
with trace("generation", metadata={"model": "gpt-4o-mini"}):
response = generate(question, docs)
return responseNote: trace is a separate function from the langsmith package, not a method of the Client object. get_embedding, retrieve and generate are functions from earlier lessons.
Evaluation runs the system on a set of questions with expected answers. The traced_query function is asynchronous, so we use aevaluate:
from langsmith import aevaluate
async def target(inputs: dict) -> dict:
"""Adapter: LangSmith passes a dict with the question and expects a dict with the answer."""
return {"answer": await traced_query(inputs["question"])}
def contains_expected(outputs: dict, reference_outputs: dict) -> bool:
"""Simple evaluator: does the answer contain the expected phrase?"""
return reference_outputs["answer"].lower() in outputs["answer"].lower()
async def evaluate_rag():
"""Evaluates the RAG system."""
# Dataset with questions and expected answers, created earlier in LangSmith
results = await aevaluate(
target,
data="rag-eval-dataset",
evaluators=[contains_expected]
)
return resultsAn evaluator is an ordinary function that returns True or False. Metrics such as faithfulness or context precision are computed the same way, only the judge is a language model.
Rate Limiting and Security
A public API without limits is an invitation to abuse. First a simple limiter with a one-minute window:
from fastapi import Depends, HTTPException
from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
import time
# Rate limiter
class RateLimiter:
"""Simple rate limiter."""
def __init__(self, requests_per_minute: int = 60):
self.rpm = requests_per_minute
self.requests: dict[str, list[float]] = {}
async def check(self, user_id: str) -> bool:
now = time.time()
minute_ago = now - 60
if user_id not in self.requests:
self.requests[user_id] = []
# Remove old requests
self.requests[user_id] = [
t for t in self.requests[user_id] if t > minute_ago
]
if len(self.requests[user_id]) >= self.rpm:
return False
self.requests[user_id].append(now)
return True
rate_limiter = RateLimiter()
security = HTTPBearer()The limiter remembers the request times of every user and removes those older than 60 seconds. It keeps data in process memory, so with several service instances move it to Redis.
Now we connect authorization and the limiter to the endpoint through Depends:
async def verify_token(credentials: HTTPAuthorizationCredentials = Depends(security)):
"""Token verification."""
# In production: JWT verification
if credentials.credentials == "invalid":
raise HTTPException(status_code=401, detail="Invalid token")
return credentials.credentials
async def rate_limit(user_id: str = Depends(verify_token)):
"""Rate limiting middleware."""
if not await rate_limiter.check(user_id):
raise HTTPException(status_code=429, detail="Too many requests")
return user_id
@app.post("/query")
async def query(request: QueryRequest, user: str = Depends(rate_limit)):
return await rag_service.query(request)HTTPBearer extracts the token from the Authorization header, and verify_token returns 401 for a bad token. Exceeding the limit ends with a 429 code. This endpoint replaces the earlier /query, so keep only one in the app, because FastAPI uses the first matching route.
Docker Deployment
Finally we package the service in a container. The Dockerfile describes the application image:
# Dockerfile
FROM python:3.11-slim
WORKDIR /app
# Install dependencies
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
# Copy code
COPY . .
# Run
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]We copy requirements.txt first and the code afterwards, so a code change does not force reinstalling dependencies.
Docker Compose starts the whole base at once: the API, Qdrant and Redis:
# docker-compose.yml
version: '3.8'
services:
rag-api:
build: .
ports:
- "8000:8000"
environment:
- OPENAI_API_KEY=${OPENAI_API_KEY}
- QDRANT_HOST=qdrant
- REDIS_HOST=redis
depends_on:
- qdrant
- redis
qdrant:
image: qdrant/qdrant:latest
ports:
- "6333:6333"
volumes:
- qdrant_data:/qdrant/storage
redis:
image: redis:alpine
ports:
- "6379:6379"Compose replaces ${OPENAI_API_KEY} with the value of the host environment variable, so the key never lands in the repository. The version key is obsolete and ignored in current Compose.
The rest of the file adds the observability layer:
prometheus:
image: prom/prometheus
ports:
- "9090:9090"
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
grafana:
image: grafana/grafana
ports:
- "3000:3000"
volumes:
qdrant_data:Prometheus collects metrics, and Grafana draws charts from them. The qdrant_data volume makes the vectors survive a container restart.
Best Practices
A list of good practices sums up all the lessons of this module:
"""
Production RAG Best Practices:
1. RETRIEVAL
- Use hybrid search (vector + keyword)
- Implement reranking
- Filter by metadata
2. CHUNKING
- Experiment with chunk sizes
- Use overlap
- Consider semantic chunking
3. CACHING
- Cache embeddings
- Cache frequent queries
- Invalidate on document updates
4. MONITORING
- Track latency at each stage
- Monitor retrieval quality
- Alert on anomalies
5. SECURITY
- Rate limiting
- Input sanitization
- PII detection and filtering
6. SCALABILITY
- Horizontal scaling API
- Sharding vector database
- Async processing
"""I recommend starting with caching and monitoring, because they pay off the fastest: a cache reduces API costs and latency, and metrics show where to look for a problem. Notice that hardcoded prompts are not on the list: in production, prompts live in configuration or in a versioning system.
Congratulations! You have learned advanced RAG and Multi-Agent systems. In the next lesson you will learn LangGraph - a framework for building complex agent workflows!
Remember: production RAG is a permanent expedition base where every tent has its job and a guard counts everyone who walks in.
Docs entries used in this lesson
Spotted a mistake in this lesson?
Check yourself
Answer the questions from this lesson. Pick an answer to see right away whether it is correct.
1. What is crucial in a production RAG system?
2. Why is cache important in production RAG?
These are 2 of 3 questions for this lesson. Solve the rest in the game.
Hands-on tasks in the game
- Code editor
Implement a production RAG endpoint
- Vertical ordering
Arrange the layers of production RAG architecture:
- Code editor
Implement a complete Simple RAG
- Click in order
Arrange the LangChain chain invocation:
- Code editor
Implement a Semantic Search Engine
- Code editor
Implement a Multi-Agent Content Pipeline