Ragas, how you know retrieval is working

Ragas, how you know retrieval is working

A system answering questions from your own documents breaks in two different ways that look identical from outside. Either retrieval failed to find the right fragment, or it found it and the model answered its own way regardless. A fix for one does nothing for the other, so without separating them, tuning is guesswork.

Ragas separates them with four measures: two scoring retrieval, two scoring the answer. It is an open source library that established the vocabulary used across this whole tool category today.

Four metrics and what they say

Splitting them into two pairs is the heart of it. If context recall is low, no prompt change will help, since the right information simply is not in what the model received. If faithfulness is low while recall is high, the problem sits in the prompt or the model rather than in the vector database.

That single observation saves the most time. Teams usually start by improving the prompt, because it is easiest, while in most deployments the retrieval stage is what fails.

The project's state

Before adding this to your dependencies, check the development pace, since it has slowed noticeably in recent months.

The library has not been deprecated or archived, the documentation works, and the Apache 2.0 licence is unchanged. The most recent release, however, is 0.4.3 from January 2026, and the last change in the repository dates to February of the same year, while community issues and pull requests keep arriving and sit unaddressed. The company behind the project also changed its name, from Exploding Gradients to Vibrant Labs, so the old repository address merely redirects and links in older material point at a stale path.

The practical conclusion comes in two parts. The metrics themselves and the way they are computed are settled, so for measurement the library serves as well today as it did before and nothing in the results described below changes. Do not count, though, on a fast response to a bug or on support for freshly released models, and before committing to this library for the long run, check the date of the latest release, since that decides whether it is still a current choice.

Your first measurement

pip install ragas

from ragas import evaluate, EvaluationDataset
from ragas.metrics import (
    Faithfulness, AnswerRelevancy, LLMContextPrecisionWithReference, LLMContextRecall
)

data = EvaluationDataset.from_list([
    {
        "user_input": "How do I download an invoice copy?",
        "retrieved_contexts": ["Invoice copies are downloaded in the billing panel...", "..."],
        "response": "You download a copy in the billing panel, Payment history section.",
        "reference": "An invoice copy is available in the billing panel.",
    },
])

result = evaluate(
    dataset=data,
    metrics=[Faithfulness(), AnswerRelevancy(),
             LLMContextPrecisionWithReference(), LLMContextRecall()],
)
print(result)

Four fields describe one case: the question, the fragments retrieval returned, the system's answer, and a reference answer. That last field is needed by only some metrics, and it is what yields the most reliable results.

Scoring happens with a language model, so every case means several calls. A hundred cases across four metrics runs to several hundred calls, and that needs pricing before you start.

How to read the scores

Scores fall between zero and one and tempt you to treat them like school marks. That is a mistake, since the absolute value matters less than the difference between runs.

On faithfulness, a value above roughly eighty five hundredths means answers stay with the context. Below seventy hundredths signals that the model regularly adds things the context did not contain, and that calls for a response rather than further tuning.

On the context metrics, thresholds depend on the task. A system answering simple questions can show high precision and sufficient recall on three fragments. A system answering questions requiring information from two documents will show lower recall, and that is often natural rather than wrong.

The practical rule: record the first measurement as a baseline and compare later ones against it. "Did the change help" is answerable; "is our system good" is not, since there is no scale shared across domains.

Generating a test set

The biggest obstacle is usually having no cases with reference answers. The library can generate them from your documents, which speeds up the start.

from ragas.testset import TestsetGenerator

generator = TestsetGenerator.from_langchain(llm=model, embedding_model=embeddings)
testset = generator.generate_with_langchain_docs(documents, testset_size=50)

Review generated questions rather than accepting them unchecked. Automation shapes questions around documents, so they sometimes sit too close to the text and do not resemble what a real user writes: tersely, with typos, and sometimes about two things at once.

A sensible arrangement is a mixed set. Half generated from documents to cover the content range, and half taken from real queries to cover how people ask. The second half finds more problems.

Include cases the system should not answer, too. A question outside the documentation's scope is a good test, since the correct answer is admitting the information is missing rather than inventing content.

Costs and sampling

Scoring with a language model costs money, so a full set on every code change is financially unrealistic.

Three things reduce that cost. The first is a cheaper scoring model, though you must check whether its results correlate with a stronger one on a sample. The second is metric choice: context precision is cheap, while faithfulness requires breaking an answer into claims and checking each, so it costs the most. The third is splitting into a small set run often and a full one run before a release.

Remember too that model scoring carries noise of its own. The same case scored twice can differ slightly, so a two hundredths difference between runs means nothing. Only differences around five hundredths across dozens of cases are a signal.

What to do with the results

Diagnosis comes down to a simple decision tree, and that is this tool's greatest practical value.

Low context recall means retrieval did not find the needed information. Check in order: whether the fragment is in the index at all, whether the splitting rule cut it in half, whether the embedding model handles your language, and whether you return enough fragments.

Low context precision with good recall means the results carry much noise. Metadata filtering helps, as does hybrid search combining vectors with keyword matching, and reranking results with a separate model.

Low faithfulness with good context is a generation side problem. The instruction should state plainly that the answer must rest solely on the supplied fragments and that missing information must be admitted. Reducing the fragment count sometimes helps too, since a model loses its way across twenty and copes with five.

Low answer relevancy with good faithfulness usually means the model answers a different question from the one asked. That often follows from question rewriting before retrieval or from an instruction encouraging elaborate answers.

Work through that tree in this order rather than starting with whatever is easiest to change. Fixing the prompt while context recall is low gains nothing, takes an afternoon, and leaves the impression that the problem is unsolvable.

How the metrics are computed

It helps to know what happens underneath, since that explains both the cost and the limits of these measures.

Faithfulness runs in two steps. First the model breaks the answer into individual claims, then it checks each against the supplied context. The score is the share of claims with support. That is where the cost comes from: an answer with eight claims is nine calls rather than one.

Answer relevancy works in reverse. The model generates questions this answer would answer well, then compares them with the actual question through embedding similarity. The practical consequence is that a rambling answer adding information beyond the question lowers this measure even when it is correct.

Context recall requires a reference answer, since it checks whether every sentence of the reference maps to a retrieved fragment. Without a reference the measure has no anchor and you must use an approximate variant, which is less reliable.

Context precision scores each retrieved fragment separately for whether it helps answer, taking order into account. A relevant fragment in first position raises the score more than the same fragment in tenth, matching how a model actually reads context.

Measuring in production

A test set shows how the system handles cases you anticipated. Production shows the cases you did not, and measuring there is worthwhile.

One limitation matters: production has no reference answers, so two of the four metrics drop out. Faithfulness and answer relevancy remain, both computed without a reference, and that suffices to detect a decline.

The practical arrangement uses sampling. Scoring ten percent of traffic gives an adequate picture at a tenth of the cost, and cases with low faithfulness enter a review queue.

That queue becomes the test set for the next cycle. It closes the loop: production supplies cases, the set keeps a fix from breaking them, and the next production measurement checks whether the fix actually worked for users.

Ragas against the alternatives

The last row matters in the light of the section on project status. That library's triad describes the same three broken links, and its development is visibly livelier, with releases from the middle of 2026. There is a difference worth knowing before switching, though: it carries no separate context recall measure, so a fragment retrieval never returned surfaces only indirectly, through a poor answer score. Recall is the measure that needs a reference answer here, and it is usually the one that settles whether to fix the index or the prompt.

These tools answer different questions and two are usually used together. This one answers where the problem sits in a retrieval system. An observability tool answers what actually happens in production and supplies cases for the set.

Note too that the metrics described here spread widely, so you will find them in other tools under the same names. The library choice therefore matters less than understanding what each measure says.

While comparing, separate two things routinely confused in conversations about the quality of retrieval systems. The first is retrieval accuracy measured without a model: did the right fragment land in the top three results. That you check with an ordinary script, for free and deterministically, so it is the place to start.

The second is answer quality, which requires a model to judge and costs money. If the first measure is low, the second has nothing to measure, because the model receives poor material. Diagnosis should therefore run from the cheap retrieval measurement to the expensive answer measurement, not the other way round.

Common mistakes

The first is measuring answer quality alone. Without context metrics you cannot tell whether to fix the prompt or retrieval, and those are entirely different jobs.

The second is treating an absolute score as a grade. A value of eighty hundredths means nothing without comparison to a previous run and without domain knowledge.

The third is a set made of automatically generated questions. Such questions sit too close to the documents and do not resemble how people ask.

The fourth is having no out of scope cases. A system that answers everything scores well in measurement and badly in practice, since questions without coverage in the documents deserve an admission of not knowing.

The fifth is drawing conclusions from differences smaller than measurement noise. Model scoring is not deterministic, so two hundredths of difference is chance rather than improvement.

The sixth is measuring once at rollout. Documents change, the vendor's model changes, and quality drops quietly, so the measurement deserves repeating on a cycle.

FAQ

What exactly is Ragas for?

For separating two causes of poor answers in a system answering from your own documents: a retrieval error and a generation error. Four metrics show which stage the problem sits in, which translates directly into what to fix.

Is Ragas free?

Yes, the library is open source and you run it yourself. You do pay for the model calls the scoring uses, and with the full metric set that is several calls per test case.

How many test cases are needed?

Thirty to fifty suffices to detect a difference between variants. Variety matters more than count: simple questions, ones requiring combining information, and ones the system should not answer.

How does Ragas differ from Promptfoo?

Ragas carries metrics fitted to retrieval systems and says which stage the problem sits in. Promptfoo compares prompt and model variants and generates adversarial probes. They work best together, since they answer different questions.

Can measurement be automated?

Yes, the library runs in a build pipeline like an ordinary test. Given the cost, a sensible split runs a small set on every change and the full one before a release, with the pass threshold set at set level rather than per case.

Documentation sits on the project site, and the source code in the GitHub repository.