GraphRAG, questions vector search cannot answer

GraphRAG, questions vector search cannot answer

Semantic similarity search answers pointed questions well: where the returns procedure is described, what the limit is on the base plan. It fails on questions spanning a whole corpus: what the main themes across these hundred reports are, who worked with whom, what recurs in last quarter's tickets.

The reason is simple. An answer to such a question lies in no single fragment, so even perfect retrieval returns five fragments from which it cannot be assembled.

GraphRAG approaches that differently: it builds an entity and relation graph from documents, clusters it into communities, summarises each, and then answers corpus wide questions using those summaries rather than fragments.

How it works

The flow has two phases of entirely different cost, and separating them from the start helps.

The indexing phase is expensive. A model walks every document and extracts entities: people, organisations, products, concepts. Then it extracts relations between them. A graph forms, which a community detection algorithm splits into clusters of related entities, and the model summarises each cluster.

The answering phase uses what was produced. A pointed question goes to local search, covering an entity and its immediate graph neighbourhood. A global question goes to community summaries, from which the model assembles a corpus wide answer.

That split is the heart of it. Local search answers questions about specifics, global search questions about the whole, and the difference lies in what the answer draws on.

Cost, the main obstacle

It deserves saying plainly: the original version is expensive. Extracting entities and relations from every document means passing the whole corpus through a model, and then summarising every community as well.

At a thousand documents that runs to hundreds of dollars; at ten thousand, to thousands. It is a one off indexing cost, and it returns on every substantial corpus change and on every model change.

The answer to that problem is variants deferring part of the work until query time. Rather than summarising every community up front, they do it for those that prove relevant to a specific question. Indexing cost then falls to a level comparable with ordinary vector search while answer quality holds.

Lighter implementations of the same idea with lower requirements also exist. When planning a deployment, start with those, since the original version is chiefly research work showing the approach works rather than the cheapest route to a result.

When it makes sense

Separate the kinds of question, since the whole decision follows from that.

Pointed questions are served by ordinary vector search and served more cheaply. If your users ask about specific procedures, values, and rules, adding a graph will not improve answers and will raise cost.

Whole corpus questions are where the graph wins. Summarising themes across a set of reports, finding recurring threads in tickets, describing how something changed over time across a document series.

Relationship questions are the second such area. Who connects to whom, through what, and how far. Vector search will not do that, since semantic similarity carries no information about relations.

The practical conclusion is to base the decision on your users' real questions. If ninety of a hundred queries are pointed, build ordinary search and consider a graph for the remaining ten. If the proportion inverts, the situation differs.

The hybrid arrangement

The commonest production solution does not choose one approach but routes a question to wherever it will be served best.

A routing layer identifies the question type and picks a path. A question about specifics goes to a vector database such as Qdrant. A question about numbers goes to an ordinary relational query. A question about the whole goes to community summaries.

That split makes economic sense too. The cheapest path serves most traffic and the expensive one runs only when needed. The reverse arrangement, sending everything through the graph, pays the highest rate for questions that do not need it.

The routing layer can be a simple classifier on a cheaper model. One extra call costs little against the difference between paths, and classification accuracy can be measured on a set of real questions.

A trial run

Before pricing the whole corpus, build a graph for a small part of it. That takes an hour and answers questions no amount of reading will settle.

pip install graphrag
graphrag init --root ./project
graphrag index --root ./project

Configuration covers the model for entity extraction, the embedding model, and the entity types to look for. That last parameter matters most on specialist documents, since by default the tool looks for organisations, people, places, and events only.

extract_graph:
  entity_types: [person, organisation, product, technology, contract]

The section name changed with the first stable release: older guides give entity_extraction, while the current release reads extract_graph. The tool also requires Python between 3.11 and 3.13, so installation fails on anything older or newer.

Running indexing across twenty documents shows three things. What one document costs, giving a basis to multiply. Which entities actually emerge, telling you whether the types fit. What the graph looks like, telling you whether relations make sense.

That trial usually leads to two or three configuration fixes after which the result is markedly better. Running the whole corpus straight away means paying full price for a version you would have to repeat anyway.

Check how global search behaves on so small a corpus too. At twenty documents the communities are few, so answers to whole corpus questions come out thin, and that is a consequence of sample size rather than a fault in the method.

Graph quality

The whole construction rests on how well the model extracted entities and relations, so it helps to know where that fails.

The first problem is the same entity under different names. A company appearing once by full name, once abbreviated, and once with a typo becomes three nodes instead of one, scattering its relations. An entity merging step is mandatory here rather than optional.

The second is entities extracted from fragments without context. A model seeing one paragraph does not know the person named is the same one mentioned three pages earlier. Larger fragments improve that at a cost.

The third is domain. A general model extracts general entities, and with specialist documents it helps to supply the entity types to look for. Without that, medical documentation yields people and dates rather than substances and interactions.

Inspect the graph after building it before considering the matter closed. A quarter of an hour reviewing entities usually reveals two or three systematic errors whose correction changes answer quality more than any prompt tuning.

For reviewing alone it is easier to load the indexing output into a graph database, since the tool writes the graph out as table files and hunting for duplicated entities in those is laborious. Neo4j fits that job well, because a Cypher query pulls out nodes with similar names, or nodes carrying no relation at all, directly. One detail helps when hunting for documentation: the project moved to calendar versioning, so there is no such thing as Neo4j 6, and material discussing a sixth major release predates the change or is a misunderstanding.

Updating the corpus

Documents change, and rebuilding the whole graph on every change is financially impossible. Settle that question before deployment rather than after the first update.

Incremental addition extracts entities from new documents only and wires them into the existing graph. It works well while new content does not change the overall picture, since communities and their summaries stay as they were.

The problem appears when new documents shift the structure. A new department, a new product line, or a change after which existing clusters stop fitting all require community detection to run again and summaries to be recomputed.

The practical answer separates the two cases. Incremental addition on every change and a full rebuild quarterly or after a substantial shift in how the corpus is organised. The latter is then planned and budgeted rather than discovered as a necessity.

Decide what happens to deleted documents too. An entity extracted from a document that no longer exists stays in the graph and influences answers, so deletion must reach the graph rather than the document store alone.

What it delivers in numbers

Assessing this approach is hard because its advantage shows only on questions of a particular type, and there it can be large.

On pointed questions the difference from ordinary retrieval usually sits within measurement noise. That is expected, since both methods reach the same fragment by different routes.

On corpus wide questions the difference is qualitative rather than quantitative. Ordinary retrieval returns five fragments and the model produces an answer based on five examples, presenting it as a picture of the whole. That is worse than no answer, since it sounds credible and misleads.

That mechanism deserves attention when assessing an existing system. If users ask corpus wide questions and the system answers fluently, check how many fragments the answer rests on. The measures covered in the piece on Ragas will not catch it, since the answer is faithful to the supplied context and only the context is unrepresentative.

The third thing is query cost. Global search passes through many community summaries, so one question costs many times an ordinary one. Variants limiting how many communities are reviewed reduce that considerably, though never to zero.

GraphRAG against the alternatives

The last row deserves attention as often the simplest answer. A corpus fitting inside a current model's context window needs no indexing: you drop it in and ask. At two hundred pages and a few questions a day that comes out cheaper than building a graph.

The boundary sits at size and frequency. A corpus exceeding the context window or questions asked hundreds of times a day shift the arithmetic towards indexing solutions.

Common mistakes

The first is building a graph without checking what users actually ask. If every question is pointed, all that work changes no answers.

The second is skipping entity merging. The same company as three nodes scatters relations and breaks community detection, so summaries cover arbitrary sets.

The third is running the original version on a large corpus without pricing it. The indexing bill can exceed a year's model budget.

The fourth is assuming a graph replaces vector search. It complements it, answering a different kind of question rather than succeeding it.

The fifth is having no update plan. Documents change, and rebuilding the whole graph on every change is financially impossible, so know from the start how new documents are added incrementally.

The sixth is judging quality from an impression across a few questions. The same measures as with ordinary retrieval, covered in the piece on Ragas, apply here identically, and without them tuning is guesswork.

FAQ

How does GraphRAG differ from ordinary RAG?

Ordinary retrieval returns fragments similar to the question, so it answers pointed questions. GraphRAG builds an entity and relation graph, clusters it into communities, and summarises them, so it also answers corpus wide questions no single fragment could answer.

What does building a graph cost?

The original version passes the whole corpus through a model several times, so at a thousand documents that is hundreds of dollars. Variants deferring summarisation until query time bring that down to a level comparable with ordinary vector search.

Does it replace a vector database?

No, it complements one. The commonest production arrangement routes pointed questions to a vector database, numeric questions to a relational query, and corpus wide questions to community summaries, since each path answers a different kind of question.

When is it not worth reaching for?

When users ask pointed questions only, when the corpus fits a model's context window and questions are few, or when the budget excludes indexing. In the first two cases simpler solutions give the same result more cheaply.

Where should I start?

By gathering thirty real user questions and splitting them into pointed and global. That proportion says whether it is worth it, and if you proceed, start with a lighter variant on a small document subset before pricing the whole.

Code and documentation sit in the project repository, and the cost reducing variant is described in a research post.