Python course Β· Module 11: RAG and Multi-Agent Systems

Vector Databases

9 min read Β·
In this lesson6

In the previous lesson we kept embeddings in a NumPy matrix. With five documents that is enough, but with a million the matrix does not fit in memory, disappears when the program restarts, and scanning it every time takes too long. We need an archive of tracks that survives the night at camp and answers in milliseconds.

Vector databases are specialized databases optimized for storing and searching embeddings. They are like a library with a magical catalog that finds similar books based on their content, not their title.

The diagram groups the popular databases by where they run:

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚              Vector Databases Landscape                  β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚                                                          β”‚
β”‚  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”‚
β”‚  β”‚   Pinecone   β”‚  β”‚    Qdrant    β”‚  β”‚   Chroma     β”‚  β”‚
β”‚  β”‚   (Cloud)    β”‚  β”‚  (Self-host) β”‚  β”‚   (Local)    β”‚  β”‚
β”‚  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜  β”‚
β”‚                                                          β”‚
β”‚  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”‚
β”‚  β”‚   Weaviate   β”‚  β”‚    Milvus    β”‚  β”‚    FAISS     β”‚  β”‚
β”‚  β”‚  (Semantic)  β”‚  β”‚  (Enterprise)β”‚  β”‚  (In-memory) β”‚  β”‚
β”‚  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜  β”‚
β”‚                                                          β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Pinecone runs only in the cloud as a managed service, Qdrant and Milvus can be hosted on your own server, Chroma works great locally for prototypes, and FAISS is a library from Meta that keeps the index in the memory of your process.

Chroma - Local Vector Database

Chroma is a local vector database for RAG applications that you install with a single pip install chromadb. First we create a client, an embedding function and a collection, the equivalent of a table:

import chromadb
from chromadb.utils import embedding_functions

# Initialize client
client = chromadb.Client()  # In-memory
# or: client = chromadb.PersistentClient(path="./chroma_db")  # Persistent

# Configure embedding function
openai_ef = embedding_functions.OpenAIEmbeddingFunction(
    api_key="your-key",
    model_name="text-embedding-3-small"
)

# Create collection
collection = client.create_collection(
    name="python_safari",
    embedding_function=openai_ef,
    metadata={"description": "Python Safari course documentation"}
)

chromadb.Client() keeps data only in memory, while PersistentClient saves it to a directory on disk. The collection calls openai_ef for every document by itself, so you do not have to compute embeddings manually. Do not type the API key into the code as in the example: in a real project read it from an environment variable.

Now we add documents with metadata and search right away:

# Add documents
collection.add(
    documents=[
        "Python is a high-level programming language",
        "RAG combines retrieval with text generation",
        "Vector databases store embeddings"
    ],
    metadatas=[
        {"topic": "python", "level": "beginner"},
        {"topic": "ai", "level": "advanced"},
        {"topic": "databases", "level": "intermediate"}
    ],
    ids=["doc1", "doc2", "doc3"]
)

# Search
results = collection.query(
    query_texts=["How to start learning programming?"],
    n_results=2,
    where={"level": "beginner"}  # Filter by metadata
)

print(results)

Every document needs a unique id. The where parameter filters by metadata before vectors are compared, so only the document with level equal to beginner passes here. The result is a dictionary of lists, one per question.

If you already have your own vectors, for example from the get_embedding function in the previous lesson, pass them in the embeddings parameter and Chroma skips the computation:

# Your own vectors instead of the collection's embedding function
collection.add(
    documents=["Elephants walk to the watering hole at dawn"],
    embeddings=[get_embedding("Elephants walk to the watering hole at dawn")],
    ids=["doc4"]
)

The vectors must have the same dimension as the rest of the collection, which is why we use the same text-embedding-3-small model.

Qdrant is a database you run on a server or in a Docker container. Working with it always has four steps: connect the client, create a collection with a vector configuration, add points and search. The first two steps look like this:

from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams, PointStruct
import numpy as np

# Connect to Qdrant
client = QdrantClient(host="localhost", port=6333)
# or: client = QdrantClient(":memory:")  # In-memory

# Create collection
client.create_collection(
    collection_name="documents",
    vectors_config=VectorParams(
        size=1536,  # Embedding size
        distance=Distance.COSINE
    )
)

VectorParams tells the database how long the vectors will be and which measure to compare them with. size must match the embedding model, here 1536 for text-embedding-3-small.

A point in Qdrant is a vector plus a payload, an arbitrary dictionary of data:

# Add points
def add_documents(texts: list[str], embeddings: list[list[float]]):
    """Adds documents to Qdrant."""
    points = [
        PointStruct(
            id=i,
            vector=embedding,
            payload={"text": text}
        )
        for i, (text, embedding) in enumerate(zip(texts, embeddings))
    ]

    client.upsert(
        collection_name="documents",
        points=points
    )

The upsert method inserts new points or overwrites existing ones with the same id, which is why running it again does not create duplicates.

In the current Python client we search with the query_points method. The older search method was marked as deprecated, and new versions of qdrant-client no longer have it:

# Search
def search(query_embedding: list[float], limit: int = 5):
    """Searches for similar documents."""
    results = client.query_points(
        collection_name="documents",
        query=query_embedding,
        limit=limit
    ).points

    return [
        {
            "text": hit.payload["text"],
            "score": hit.score
        }
        for hit in results
    ]

query_points returns an object with a points list, and every hit has a score and a payload.

A filter works like where in Chroma, only with a more formal syntax:

# Filtering
from qdrant_client.models import Filter, FieldCondition, MatchValue

filtered_results = client.query_points(
    collection_name="documents",
    query=query_embedding,
    query_filter=Filter(
        must=[
            FieldCondition(
                key="category",
                match=MatchValue(value="python")
            )
        ]
    ),
    limit=5
).points

Watch out for a trap: add_documents stores only text in the payload, so a filter on category finds nothing until you add that field to the payload. query_embedding is the question vector you have already computed.

FAISS is not a server but a library. The index lives in the memory of your process. Let's wrap it in a class that also remembers the document texts:

import faiss
import numpy as np
from dataclasses import dataclass

@dataclass
class FAISSIndex:
    """Wrapper for FAISS."""

    dimension: int
    index: faiss.Index = None
    documents: list[str] = None

    def __post_init__(self):
        # Different index types
        # Flat - exact, slower
        self.index = faiss.IndexFlatL2(self.dimension)

        # IVF - faster, approximate
        # quantizer = faiss.IndexFlatL2(self.dimension)
        # self.index = faiss.IndexIVFFlat(quantizer, self.dimension, 100)

        self.documents = []

    def add(self, embeddings: np.ndarray, documents: list[str]):
        """Adds vectors to the index."""
        embeddings = np.array(embeddings).astype('float32')
        self.index.add(embeddings)
        self.documents.extend(documents)

IndexFlatL2 compares the question with every vector, so the result is exact. The commented-out IndexIVFFlat is faster but approximate, and it needs training with the train method before you add vectors. FAISS accepts only float32, hence the conversion in add.

Searching returns the distances and the numbers of the nearest neighbours:

    def search(self, query_embedding: np.ndarray, k: int = 5) -> list[tuple[str, float]]:
        """Searches for k nearest neighbors."""
        query = np.array([query_embedding]).astype('float32')
        distances, indices = self.index.search(query, k)

        results = []
        for idx, dist in zip(indices[0], distances[0]):
            if 0 <= idx < len(self.documents):
                results.append((self.documents[idx], float(dist)))

        return results

A smaller L2 distance means greater similarity, the opposite of cosine. When you ask for more results than there are vectors, FAISS returns the index -1, which is why we check 0 <= idx.

A test on random vectors shows the mechanism itself:

# Usage example
faiss_index = FAISSIndex(dimension=384)

# Add documents
embeddings = np.random.rand(100, 384).astype('float32')
documents = [f"Document {i}" for i in range(100)]
faiss_index.add(embeddings, documents)

# Search
query = np.random.rand(384).astype('float32')
results = faiss_index.search(query, k=5)

In real code, instead of random numbers you plug in embeddings from a 384-dimensional model.

Pinecone - Managed Vector Database

Pinecone is a cloud service: you do not install a server, you create an index through the API. When creating it you specify the dimension, the metric and the region:

from pinecone import Pinecone, ServerlessSpec

# Initialize
pc = Pinecone(api_key="your-key")

# Create index
pc.create_index(
    name="python-safari",
    dimension=1536,
    metric="cosine",
    spec=ServerlessSpec(
        cloud="aws",
        region="us-east-1"
    )
)

# Connect to index
index = pc.Index("python-safari")

ServerlessSpec means a serverless index, billed by usage.

Upsert takes a list of dictionaries with an id, a vector and metadata:

# Upsert (insert/update)
index.upsert(
    vectors=[
        {
            "id": "doc1",
            "values": [0.1, 0.2, ...],  # 1536 values
            "metadata": {
                "text": "Python is a programming language",
                "category": "programming",
                "level": 1
            }
        }
    ],
    namespace="tutorials"
)

The notation [0.1, 0.2, ...] is shorthand: in Python the three dots are the Ellipsis object, so in real code you put a full list of 1536 numbers. A namespace splits the index into independent compartments.

A query combines a vector with a metadata filter:

# Search
results = index.query(
    vector=[0.1, 0.2, ...],
    top_k=10,
    include_metadata=True,
    namespace="tutorials",
    filter={
        "category": {"$eq": "programming"},
        "level": {"$lte": 3}
    }
)

# Statistics
stats = index.describe_index_stats()
print(f"Number of vectors: {stats['total_vector_count']}")

The $eq and $lte operators mean "equal" and "less than or equal", and describe_index_stats reports how many vectors are in the index.

Hybrid Search - Combining Vectors with BM25

Vectors are great at catching meaning, but they can miss an exact name, such as a model number. BM25 is a classic keyword search algorithm. Hybrid search combines both scores:

from rank_bm25 import BM25Okapi
import numpy as np

class HybridSearch:
    """Combines semantic search with keyword search."""

    def __init__(self, documents: list[str], embeddings: np.ndarray):
        self.documents = documents
        self.embeddings = embeddings

        # BM25 for keyword search
        tokenized = [doc.lower().split() for doc in documents]
        self.bm25 = BM25Okapi(tokenized)

The constructor splits every document into words and builds a BM25Okapi index from them, and it receives the embeddings ready-made.

The search method normalizes both scores to the 0-1 range and mixes them with the alpha weight:

    def search(
        self,
        query: str,
        query_embedding: np.ndarray,
        alpha: float = 0.5,  # Semantic search weight
        top_k: int = 5
    ) -> list[tuple[str, float]]:
        """Hybrid search with configurable weights."""

        # Semantic search scores
        semantic_scores = np.dot(self.embeddings, query_embedding)
        semantic_scores /= np.linalg.norm(self.embeddings, axis=1)
        semantic_scores /= np.linalg.norm(query_embedding)

        # Normalize to [0, 1]
        semantic_scores = (semantic_scores - semantic_scores.min()) / (semantic_scores.max() - semantic_scores.min())

        # BM25 scores
        bm25_scores = np.array(self.bm25.get_scores(query.lower().split()))
        if bm25_scores.max() > 0:
            bm25_scores = bm25_scores / bm25_scores.max()

        # Hybrid score
        hybrid_scores = alpha * semantic_scores + (1 - alpha) * bm25_scores

        # Top K
        top_indices = np.argsort(hybrid_scores)[::-1][:top_k]

        return [(self.documents[i], hybrid_scores[i]) for i in top_indices]

alpha=1 is pure semantic search, alpha=0 is pure BM25. I recommend starting at 0.5 and tuning the weight on your own test questions.

Vector databases are the infrastructure of every RAG system. For a prototype take Chroma, for production Qdrant or Pinecone. In the next lesson you will learn LlamaIndex - a framework that simplifies building RAG applications and connects to these databases in a single line.

Remember: a vector database is the camp's archive of tracks, which finds the most similar prints from a single footprint.

Docs entries used in this lesson

Spotted a mistake in this lesson?

Check yourself

Answer the questions from this lesson. Pick an answer to see right away whether it is correct.

  1. 1. Which of the following is a popular vector database?

  2. 2. What is Chroma in the context of vector databases?

Hands-on tasks in the game

  • Horizontal ordering

    Arrange the elements:

  • Code editor

    Implement adding documents to Chroma

  • Horizontal ordering

    Arrange the elements:

  • Vertical ordering

    Arrange the steps for working with Qdrant in the correct order:

  • Click in order

    Arrange adding to vector store:

  • Vertical ordering

    Arrange the steps for creating a FAISS index:

  1. QuestionVector Database
  2. QuestionChroma DB
  3. Put in orderImport ChromaDB
  4. Coding taskChroma Collection
  5. Put in orderChroma query
  6. Put in orderQdrant workflow
  7. Click in orderVector Store Add
  8. Put in orderFAISS Index

Useful articles