RAG - Extended Memory for AI

Welcome to the penultimate stage of Python Safari! Just as large mammals evolved by developing bigger brains and better memory, language models are evolving thanks to RAG (Retrieval-Augmented Generation) technology. This is a groundbreaking technique that gives AI access to external knowledge!

The Problem with Traditional LLMs

Language models have fundamental limitations, much like animals adapted only to a single environment:

Limitations of Base LLMs:

1. Knowledge cutoff - knowledge frozen at training time
2. Hallucinations - generating false information
3. Lack of company context - they don't know your documents
4. High fine-tuning costs - adapting the model is expensive

What is RAG?

RAG is an architecture that combines information retrieval with text generation:

1┌─────────────────────────────────────────────────────────┐
2│                    RAG Pipeline                          │
3├─────────────────────────────────────────────────────────┤
4│                                                          │
5│  User question                                           │
6│         │                                                │
7│         ▼                                                │
8│  ┌─────────────────┐                                    │
9│  │   Embedding     │  ← Convert to vector               │
10│  └────────┬────────┘                                    │
11│           │                                              │
12│           ▼                                              │
13│  ┌─────────────────┐     ┌─────────────────┐           │
14│  │ Vector Search   │────▶│ Vector Database │           │
15│  └────────┬────────┘     └─────────────────┘           │
16│           │                                              │
17│           ▼                                              │
18│  ┌─────────────────┐                                    │
19│  │ Retrieved Docs  │  ← Top-K similar documents         │
20│  └────────┬────────┘                                    │
21│           │                                              │
22│           ▼                                              │
23│  ┌─────────────────┐                                    │
24│  │   LLM + Context │  ← Generate with context           │
25│  └────────┬────────┘                                    │
26│           │                                              │
27│           ▼                                              │
28│      Answer                                              │
29└─────────────────────────────────────────────────────────┘

Basic RAG Implementation

1from openai import OpenAI
2import numpy as np
3
4client = OpenAI()
5
6# Simple knowledge base
7knowledge_base = [
8    "Python Safari is a Python programming course with 12 modules.",
9    "RAG combines document retrieval with text generation by LLMs.",
10    "Embeddings are vector representations of text in semantic space.",
11    "Vector databases store and search embeddings efficiently.",
12    "LlamaIndex is a framework for building RAG applications.",
13    "CrewAI enables the creation of multi-agent systems."
14]
15
16def get_embedding(text: str) -> list[float]:
17    """Generates an embedding for the text."""
18    response = client.embeddings.create(
19        model="text-embedding-3-small",
20        input=text
21    )
22    return response.data[0].embedding
23
24def cosine_similarity(vec1: list, vec2: list) -> float:
25    """Computes cosine similarity of two vectors."""
26    vec1, vec2 = np.array(vec1), np.array(vec2)
27    return np.dot(vec1, vec2) / (np.linalg.norm(vec1) * np.linalg.norm(vec2))
28
29def retrieve_relevant_docs(query: str, docs: list, top_k: int = 3) -> list[str]:
30    """Retrieves the most similar documents."""
31    query_embedding = get_embedding(query)
32
33    similarities = []
34    for doc in docs:
35        doc_embedding = get_embedding(doc)
36        similarity = cosine_similarity(query_embedding, doc_embedding)
37        similarities.append((doc, similarity))
38
39    # Sort by similarity
40    similarities.sort(key=lambda x: x[1], reverse=True)
41
42    return [doc for doc, _ in similarities[:top_k]]
43
44def rag_query(question: str) -> str:
45    """Full RAG pipeline."""
46    # 1. Retrieve - find relevant documents
47    relevant_docs = retrieve_relevant_docs(question, knowledge_base)
48
49    # 2. Augment - build prompt with context
50    context = "\n".join(relevant_docs)
51
52    prompt = f"""Answer the question using ONLY the context below.
53If you cannot answer based on the context, say "I don't know".
54
55Context:
56{context}
57
58Question: {question}
59"""
60
61    # 3. Generate - produce the answer
62    response = client.chat.completions.create(
63        model="gpt-4o-mini",
64        messages=[{"role": "user", "content": prompt}]
65    )
66
67    return response.choices[0].message.content
68
69# Test
70answer = rag_query("What is RAG?")
71print(answer)

Chunking - Splitting Documents

Long documents need to be split into smaller pieces (chunks):

1from typing import Generator
2
3def simple_chunker(text: str, chunk_size: int = 500, overlap: int = 100) -> Generator[str, None, None]:
4    """Simple function for splitting text into chunks."""
5    start = 0
6    while start < len(text):
7        end = start + chunk_size
8        chunk = text[start:end]
9        yield chunk
10        start = end - overlap  # Overlap to preserve context
11
12def sentence_chunker(text: str, sentences_per_chunk: int = 5) -> list[str]:
13    """Splits text into chunks by sentences."""
14    import re
15
16    # Simple sentence segmentation
17    sentences = re.split(r'(?<=[.!?])\s+', text)
18
19    chunks = []
20    for i in range(0, len(sentences), sentences_per_chunk):
21        chunk = ' '.join(sentences[i:i + sentences_per_chunk])
22        chunks.append(chunk)
23
24    return chunks
25
26# Usage example
27document = """
28Python is a high-level programming language. It was created by
29Guido van Rossum. It is very popular in Data Science and Machine Learning.
30RAG is a technique combining retrieval with generation. It allows LLMs
31to use external knowledge sources. It is crucial for enterprise AI.
32"""
33
34chunks = sentence_chunker(document, sentences_per_chunk=2)
35for i, chunk in enumerate(chunks):
36    print(f"Chunk {i}: {chunk[:50]}...")

RAG Quality Metrics

1from dataclasses import dataclass
2
3@dataclass
4class RAGMetrics:
5    """Metrics for evaluating a RAG system."""
6
7    def relevance_score(self, query: str, retrieved_doc: str) -> float:
8        """Is the document relevant to the question?"""
9        # In practice we use an LLM for evaluation
10        pass
11
12    def faithfulness_score(self, answer: str, context: str) -> float:
13        """Is the answer grounded in the context?"""
14        # Checks for hallucinations
15        pass
16
17    def answer_correctness(self, answer: str, ground_truth: str) -> float:
18        """Is the answer correct?"""
19        # Comparison with ground truth
20        pass
21
22# Simple evaluation example
23def evaluate_retrieval(query: str, retrieved: list[str], relevant: list[str]) -> dict:
24    """Computes precision and recall for retrieval."""
25    retrieved_set = set(retrieved)
26    relevant_set = set(relevant)
27
28    true_positives = len(retrieved_set & relevant_set)
29
30    precision = true_positives / len(retrieved_set) if retrieved_set else 0
31    recall = true_positives / len(relevant_set) if relevant_set else 0
32
33    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
34
35    return {
36        "precision": precision,
37        "recall": recall,
38        "f1": f1
39    }

RAG is the foundation of modern enterprise AI applications. In the next lesson you will learn about embeddings - the heart of the entire search system!