LlamaIndex, a framework built around documents

LlamaIndex, a framework built around documents

LlamaIndex does one thing properly: it connects a language model to your data. It loads documents, splits them, indexes them, and then answers questions from them. The framework is open source and free, while the paid layer handles parsing difficult files.

How it differs from LangChain

The two overlap and either can build the same thing, but they start from different places.

LangChain starts from the model and the question of how to wire it to tools, memory, and control flow. LlamaIndex starts from documents and the question of how to organise them so the model receives the right chunks. The difference shows in defaults: the first gives you more building blocks and demands decisions, the second ships sensible retrieval defaults you then adjust gradually.

The practical split runs like this. If document search is the core of the product, start with LlamaIndex, because you reach a working result faster. If the core is an agent making decisions and calling tools, LangChain or LangGraph fit better. Plenty of projects use both, since they coexist in one codebase.

Concepts worth knowing up front

Four names recur throughout the documentation, and examples are hard to read without them.

A document is a loaded file together with metadata. A node is a chunk of a document after splitting, meaning the unit the model actually receives. An index is the structure that finds those nodes, usually vector based. A query engine ties a retriever to the model and returns an answer along with its sources.

It helps to grasp early that the node is the unit everything rests on. It goes into the vector database, it comes back from retrieval, and it lands in the prompt. If splitting cuts a sentence in half or separates a table header from its rows, no later layer repairs that. So the first hours on a system are better spent looking at a dozen generated nodes than on picking a model.

That last element gets undervalued. An answer carries references to the nodes it came from, so you can show the user a quotation and a link to the document. In corporate deployments that is often a precondition, because an answer without a source is useless to somebody who has to verify it.

Your first RAG

pip install llama-index
export OPENAI_API_KEY=...

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader

documents = SimpleDirectoryReader("./data").load_data()
index = VectorStoreIndex.from_documents(documents)

engine = index.as_query_engine(similarity_top_k=4)
answer = engine.query("What is the return period for goods?")

print(answer)
for source in answer.source_nodes:
    print(source.metadata["file_name"], round(source.score, 3))

Four lines produce working retrieval, which can mislead. This version keeps the index in process memory, so it disappears on restart and is unfit for production. For durable storage you attach a vector database such as Pinecone or Postgres with a vector extension from Supabase.

from llama_index.vector_stores.postgres import PGVectorStore
from llama_index.core import StorageContext

store = PGVectorStore.from_params(
    database="data", host="localhost", port=5432,
    user="postgres", password=password, table_name="chunks", embed_dim=1536
)
context = StorageContext.from_defaults(vector_store=store)
index = VectorStoreIndex.from_documents(documents, storage_context=context)

llama-index on its own pulls in neither database integrations nor the cloud client. The example above needs the llama-index-vector-stores-postgres package, and LlamaParse in the next section needs llama-cloud-services. An import without the install fails with an error easily mistaken for a misspelled class name.

Parsing, where the real problem sits

Most RAG projects do not fall over on model choice, they fall over on document loading. A PDF with a financial table, a scanned contract, a slide deck with charts, or a two column layout turns, after naive loading, into text whose order makes no sense.

The symptom is distinctive: the model answers confidently and wrongly, because it received a chunk where numbers from one column merged with labels from another. The fault lies in step one while everybody hunts for it in the prompt.

LlamaParse is the commercial answer to that problem. It recognises page layout, preserves table structure, and returns markdown rather than flat text.

On the open source side Unstructured solves the same problem, recognising file type and breaking it into elements while preserving structure. The library edition costs nothing and runs on your own machines, which can be decisive for documents that cannot leave the company.

from llama_cloud_services import LlamaParse

parser = LlamaParse(result_type="markdown")
documents = parser.load_data("./contract.pdf")

Markdown output carries a second benefit independent of reading quality. Headings and tables recorded structurally let you split a document by section rather than by character count, so a chunk matches a logical unit instead of an arbitrary slice. For technical documentation and terms of service that usually helps more than any change on the retrieval side.

Before paying for it, test your files with the free reader. Text documents generated from an editor usually load correctly and need nothing more. Only scans and files with elaborate layouts justify the cost.

Retrieval strategies

The default returns the four most similar chunks and covers many cases. When it does not, several directions are open, each solving a different problem.

An implementation order that usually pays: fix document splitting first, add reranking next, complicate the architecture last. Changing chunk size often beats three more layers stacked on retrieval.

Workflows, event driven agents

Once a process stops being a single query, LlamaIndex offers Workflows: steps reacting to events, each able to emit further ones. The programming model resembles a message queue more than a graph.

That deserves more space as a subject of its own, since the event model changes how you think about a process. LlamaIndex Workflows covers it in more depth, along with parallel steps, suspension points, and resuming a run after a failure.

from llama_index.core.workflow import Workflow, step, StartEvent, StopEvent, Event

class FoundEvent(Event):
    chunks: list

class QuestionHandler(Workflow):
    @step
    async def search(self, ev: StartEvent) -> FoundEvent:
        return FoundEvent(chunks=retriever.retrieve(ev.question))

    @step
    async def answer(self, ev: FoundEvent) -> StopEvent:
        return StopEvent(result=model.complete(build_prompt(ev.chunks)))

The advantage is natural parallelism: if two steps react to the same event, they run concurrently. The drawback is harder tracing, since order is not written down explicitly but follows from event types.

The agent layer is in beta and carries no charge of its own. You pay for the modules an agent uses, meaning parsing and extraction.

Updating documents without corrupting the index

The first version of a system usually assumes documents are static. A month later they are not, and trouble begins, because two versions of the same terms of service now sit side by side in the index.

The fix is an identifier derived from content rather than a random one. When a node receives an id built from the file name and chunk number, reloading the document overwrites the old entries instead of adding new ones.

from llama_index.core.ingestion import IngestionPipeline
from llama_index.core.node_parser import SentenceSplitter

pipeline = IngestionPipeline(
    transformations=[SentenceSplitter(chunk_size=800, chunk_overlap=120), embedding],
    vector_store=store,
    docstore=docstore
)
pipeline.run(documents=documents)

A pipeline with a document store compares a content hash against what is already saved and processes only what changed. With a thousand files of which five change daily, the difference in cost and time is decisive.

Give deletion its own path. A file removed at the source does not vanish from the index by itself, so without deliberately deleting the nodes the model will keep quoting a document that no longer exists. That is especially awkward with personal data and with withdrawn documents.

It also pays to record the ingestion date and document version in metadata. Then, when answers contradict each other, you can establish where the older chunk came from instead of guessing.

Cloud layer pricing

The framework costs nothing. The cloud bills in credits, and a thousand credits costs 1.25 USD, the same in the North American and European regions.

New accounts receive ten thousand credits a month, worth 12.50 USD, which covers a prototype. Processing ten thousand pages in cost effective mode consumes thirty thousand credits, so it costs 37.50 USD once, since you pay for ingestion rather than for later queries.

The credit count is only half the picture, because whether you can buy more depends on the plan.

A free account simply stops processing once the allowance runs out, since pay as you go does not exist at that level. The first step up is therefore 50 USD a month rather than a smooth overage charge.

When estimating, count the first ingestion and the reprocessing that follows a change in splitting strategy. Changing chunk size means recomputing embeddings, and with scans sometimes reparsing too, so experiments on a large corpus carry a price. The cache helps here: a file LlamaParse has already processed bills only the extraction step on a later extract, fifteen credits instead of sixty in premium mode. The result stays cached for just 48 hours and caching can be turned off, so experiments spread over a week pay the full rate. Test on a sample of a few hundred pages and run the full pass only once the settings are chosen.

That one off nature matters when planning. Parsing cost lands at indexing time and on document updates, while the daily bill comes from embeddings and model calls.

LlamaIndex against the alternatives

That last row deserves attention, since it gets undervalued. Loading files, computing embeddings, and querying a vector database is about a hundred lines of code. A framework starts paying off with varied file formats and with the need to change retrieval strategy without rewriting everything. It is also worth checking whether anybody still develops the library: Embedchain promised exactly the same in a handful of lines, stopped at a March 2025 release, and its repository now redirects to a project about conversation memory, an entirely different problem.

How to tell whether it works at all

A RAG system without measurement looks fine in a demo and fails on real questions. Evaluation splits into two layers, and they have to be separated because each is measured differently.

The first is retrieval accuracy. You take thirty real questions, mark for each the document that should appear in the results, and count the share of cases where it did. That number does not depend on the model, so you measure it once and compare it after every change to document splitting.

The second is answer quality given correct context. Here you check whether the model stays within the supplied chunks or embellishes. The simplest approach is reading thirty answers by hand and counting those containing information absent from the sources.

def hit_rate(dataset, k=4):
    hits = sum(
        1 for question, expected in dataset
        if expected in [n.metadata["file_name"] for n in retriever.retrieve(question)[:k]]
    )
    return hits / len(dataset)

Collect questions from real user conversations rather than inventing them at a desk. A question written by somebody who knows the documentation reads like its table of contents and matches too easily, inflating the score and saying nothing about real usage.

Keep the set in your repository and run it after every change: a different embedding model, a different chunk size, added reranking. Without it, every discussion about whether the new version is better ends in trading impressions.

Common mistakes

The first is indexing without metadata. A chunk carrying no source, date, or department cannot later be filtered or shown to the user with a link.

The second is an in memory index in production. It works until the first restart, after which the application either recomputes everything or claims to know nothing.

The third is having no document update strategy. An amended contract added again leaves the old version in the index, so the model receives two contradictory answers and picks one at random.

The fourth is judging the system by model answers rather than retrieval accuracy. Measure first how often the right chunk reaches the results, because without that, prompt tweaking is guesswork.

The fifth is applying one default chunk size to every document type. Terms of service split on clauses, a call transcript on turns, technical documentation on sections.

FAQ

Is LlamaIndex free?

The framework is open source and free, commercially included. The cloud layer for parsing and extraction is paid, billed in credits where a thousand credits costs 1.25 USD. The Free plan grants ten thousand credits a month with no way to buy more, and the first paid plan costs 50 USD a month for forty thousand credits.

LlamaIndex or LangChain?

For document search you reach a result faster with LlamaIndex, since its defaults in that area are better. For agents making decisions and calling tools, LangChain fits better. Both can live in one project.

Does it work with models other than OpenAI?

Yes, it supports models from various vendors as well as local models, including those pulled from Hugging Face. Switching comes down to swapping the model object and the embedding model in configuration.

Do I have to use LlamaParse?

No, the framework loads files with free readers. LlamaParse earns its place on scans, tables, and multi column layouts where plain loading destroys structure. Test your own files first, since text documents usually need nothing more.

Is there a TypeScript version?

Yes, a TypeScript counterpart exists and covers the core RAG scenarios. New features land in the Python version first, so for unusual integrations check whether the equivalent already exists.

Documentation sits at developers.llamaindex.ai, and cloud pricing on llamaindex.ai.