Python course ยท Module 11: RAG and Multi-Agent Systems

LlamaIndex - RAG Framework

7 min read ยท
In this lesson8

In the three previous lessons we wrote chunking, embeddings, search and prompt assembly ourselves. That is good tracking school, but on a real expedition nobody sews their own tent before leaving. You take proven gear and focus on the route.

LlamaIndex is a framework for building RAG applications. It simplifies the whole process - from loading documents to generating answers - and provides ready-made building blocks for every step we have written by hand so far.

LlamaIndex Basics

The whole pipeline from the first lesson fits into a few lines. The Settings object globally sets the language model and the embedding model, and SimpleDirectoryReader reads all files from a directory:

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Settings
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding

# Global configuration
Settings.llm = OpenAI(model="gpt-4o-mini", temperature=0)
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")

# Load documents
documents = SimpleDirectoryReader("./docs").load_data()
print(f"Loaded {len(documents)} documents")

# Create index
index = VectorStoreIndex.from_documents(documents)

# Query engine
query_engine = index.as_query_engine()

# Query
response = query_engine.query("What is RAG?")
print(response)

The chain is always the same: SimpleDirectoryReader and .load_data() load the documents, VectorStoreIndex.from_documents splits them into chunks, computes embeddings and builds the index, and .as_query_engine() turns the index into a question engine. Under the hood exactly what we wrote ourselves happens: retrieve, augment, generate. Only the level you work at has changed.

Document Loaders

Documents rarely come in a single format. LlamaIndex has loaders for text, files and web pages:

from llama_index.core import Document
from llama_index.readers.web import SimpleWebPageReader
from llama_index.readers.file import PDFReader, DocxReader

# From text
doc = Document(text="This is sample text to be indexed.")

# From files
pdf_reader = PDFReader()
pdf_docs = pdf_reader.load_data(file="document.pdf")

# From web page
web_reader = SimpleWebPageReader()
web_docs = web_reader.load_data(urls=["https://example.com"])

# From multiple sources
from llama_index.core import SimpleDirectoryReader

reader = SimpleDirectoryReader(
    input_dir="./data",
    recursive=True,
    required_exts=[".pdf", ".docx", ".txt", ".md"]
)
documents = reader.load_data()

Document is the basic data unit: text plus metadata. PDFReader and SimpleWebPageReader come from separate packages (llama-index-readers-file, llama-index-readers-web), so you have to install them. SimpleDirectoryReader with recursive=True also walks subdirectories and takes only the extensions listed in required_exts.

Node Parsing (Chunking)

In LlamaIndex a document chunk is called a node. You can order the splitting strategies from the simplest to the most advanced: cutting by characters, like our simple_chunker from the first lesson, sentence splitting, semantic splitting and the hierarchical parser. We start with sentences:

from llama_index.core.node_parser import (
    SentenceSplitter,
    SemanticSplitterNodeParser,
)

# Simple sentence splitting
sentence_parser = SentenceSplitter(
    chunk_size=512,
    chunk_overlap=50
)
nodes = sentence_parser.get_nodes_from_documents(documents)

SentenceSplitter cuts the text into chunks of up to 512 tokens while trying not to break sentences, and chunk_overlap is the overlap you already know.

Semantic splitting goes further: it compares the embeddings of neighbouring sentences and cuts where the meaning clearly changes:

# Semantic splitting
semantic_parser = SemanticSplitterNodeParser(
    buffer_size=1,
    breakpoint_percentile_threshold=95,
    embed_model=Settings.embed_model
)
semantic_nodes = semantic_parser.get_nodes_from_documents(documents)

breakpoint_percentile_threshold=95 means a cut happens only at the 5% largest jumps in meaning. It is more expensive, because it needs an embedding of every sentence.

The hierarchical parser creates several levels of chunks at once, from large to small:

# Hierarchical splitting
from llama_index.core.node_parser import HierarchicalNodeParser

hierarchical_parser = HierarchicalNodeParser.from_defaults(
    chunk_sizes=[2048, 512, 128]
)
hierarchical_nodes = hierarchical_parser.get_nodes_from_documents(documents)

Small chunks (128 tokens) match well in search, while large ones (2048) give the model a wider context. The input documents do not change, only the way they are cut does.

Retriever Modes

A retriever is the scout that brings back matching chunks. You can configure it manually and add post-processing:

from llama_index.core.retrievers import VectorIndexRetriever
from llama_index.core.query_engine import RetrieverQueryEngine
from llama_index.core.postprocessor import SimilarityPostprocessor

# Basic retriever
retriever = VectorIndexRetriever(
    index=index,
    similarity_top_k=5
)

# With postprocessing
query_engine = RetrieverQueryEngine(
    retriever=retriever,
    node_postprocessors=[
        SimilarityPostprocessor(similarity_cutoff=0.7)
    ]
)

similarity_top_k=5 fetches five chunks, and SimilarityPostprocessor drops the ones with similarity below 0.7, so weak hits do not clutter the prompt.

The hybrid search from the previous lesson is ready here too. Note: in current versions BM25Retriever lives in the separate llama-index-retrievers-bm25 package, not in llama_index.core:

# Hybrid retriever
from llama_index.retrievers.bm25 import BM25Retriever  # pip install llama-index-retrievers-bm25
from llama_index.core.retrievers import QueryFusionRetriever

vector_retriever = index.as_retriever(similarity_top_k=5)
bm25_retriever = BM25Retriever.from_defaults(nodes=nodes, similarity_top_k=5)

hybrid_retriever = QueryFusionRetriever(
    retrievers=[vector_retriever, bm25_retriever],
    similarity_top_k=5,
    num_queries=1,
)

QueryFusionRetriever merges the results of both scouts into one list. num_queries=1 means we use only the original question, without extra variants generated by the LLM.

Query Transformations

Sometimes the problem lies in the question itself. A complex question is worth breaking into smaller ones, and a short question can be rewritten to match the documents better:

from llama_index.core.query_engine import SubQuestionQueryEngine
from llama_index.core.tools import QueryEngineTool

# Sub-question engine - breaks questions into smaller ones
tools = [
    QueryEngineTool.from_defaults(
        query_engine=index.as_query_engine(),
        name="documentation",
        description="Contains project documentation"
    )
]

sub_question_engine = SubQuestionQueryEngine.from_defaults(
    query_engine_tools=tools
)

# HyDE - Hypothetical Document Embeddings
from llama_index.core.indices.query.query_transform import HyDEQueryTransform
from llama_index.core.query_engine import TransformQueryEngine

hyde = HyDEQueryTransform(include_original=True)
hyde_query_engine = TransformQueryEngine(
    index.as_query_engine(),
    query_transform=hyde
)

SubQuestionQueryEngine breaks a question into sub-questions and sends each one to the right tool, here a single one named "documentation". HyDE first asks the model to write a hypothetical answer and searches for documents similar to it, because an answer is often closer to the documents than the question itself. include_original=True keeps the original question as well.

Chat Engine

A query engine does not remember previous questions. For a conversation we use a chat engine with memory:

from llama_index.core.chat_engine import CondenseQuestionChatEngine
from llama_index.core.memory import ChatMemoryBuffer

# Chat memory
memory = ChatMemoryBuffer.from_defaults(token_limit=3000)

# Chat engine with history
chat_engine = index.as_chat_engine(
    chat_mode="condense_question",
    memory=memory,
    verbose=True
)

# Conversation
response1 = chat_engine.chat("What is Python?")
print(response1)

response2 = chat_engine.chat("What are its applications?")
print(response2)

# Reset memory
chat_engine.reset()

The condense_question mode rewrites a follow-up question such as "What are its applications?" into a standalone question using the history, and only then searches the index. reset() clears the memory. In the newest versions ChatMemoryBuffer is marked as deprecated in favour of the Memory class from the same module, but it still works.

Vector Database Integration

By default the index lives in memory. To make it survive a restart, we connect a vector database from the previous lesson. This is what Chroma looks like:

# Chroma
from llama_index.vector_stores.chroma import ChromaVectorStore
import chromadb

chroma_client = chromadb.PersistentClient(path="./chroma_db")
chroma_collection = chroma_client.get_or_create_collection("llama_index")

vector_store = ChromaVectorStore(chroma_collection=chroma_collection)
index = VectorStoreIndex.from_vector_store(vector_store)

from_vector_store opens an index over a collection that already contains data. To write new documents into it, pass vector_store through a StorageContext to from_documents.

Qdrant and Pinecone connect in exactly the same way, only the store class changes:

# Qdrant
from llama_index.vector_stores.qdrant import QdrantVectorStore
from qdrant_client import QdrantClient

qdrant_client = QdrantClient(host="localhost", port=6333)
vector_store = QdrantVectorStore(
    client=qdrant_client,
    collection_name="llama_index"
)

# Pinecone
from llama_index.vector_stores.pinecone import PineconeVectorStore
from pinecone import Pinecone

pc = Pinecone(api_key="...")
pinecone_index = pc.Index("llama-index")
vector_store = PineconeVectorStore(pinecone_index=pinecone_index)

The rest of the code, meaning the query engine, retrievers and chat engine, stays unchanged, and that is the biggest advantage of this abstraction layer.

Evaluation

Finally we check whether the answers are trustworthy. LlamaIndex has evaluators that use a language model as a judge:

from llama_index.core.evaluation import (
    FaithfulnessEvaluator,
    RelevancyEvaluator,
    CorrectnessEvaluator
)

# Evaluators
faithfulness = FaithfulnessEvaluator()
relevancy = RelevancyEvaluator()

# Evaluate response
query = "What is RAG?"
response = query_engine.query(query)

faithfulness_result = faithfulness.evaluate_response(response=response)
print(f"Faithfulness: {faithfulness_result.passing}")

relevancy_result = relevancy.evaluate_response(query=query, response=response)
print(f"Relevancy: {relevancy_result.passing}")

FaithfulnessEvaluator checks whether the answer follows from the retrieved context, so it catches hallucinations. RelevancyEvaluator checks whether the answer and the context match the question. The passing field is either True or False.

I recommend starting with SentenceSplitter and a simple query engine, and reaching for HyDE or hybrid search only when evaluation shows a concrete problem.

LlamaIndex is a complete framework for RAG. In the next lesson you will learn about multi-agent systems - when one agent is not enough!

Remember: LlamaIndex is ready-made expedition gear that lets you focus on the route instead of sewing the tent.

Docs entries used in this lesson

Spotted a mistake in this lesson?

Check yourself

Answer the questions from this lesson. Pick an answer to see right away whether it is correct.

  1. 1. What is LlamaIndex used for?

  2. 2. Which LlamaIndex class is used to load documents from a directory?

Hands-on tasks in the game

  • Code editor

    Implement a basic Q&A system with LlamaIndex

  • Horizontal ordering

    Arrange the basic LlamaIndex pipeline:

  • Vertical ordering

    Sort chunking strategies from simplest to most advanced:

  1. QuestionLlamaIndex
  2. QuestionDocument Loader
  3. Coding taskLlamaIndex Query Engine
  4. Put in orderLlamaIndex pipeline
  5. Put in orderChunking strategies

Useful articles