TruLens, the RAG triad and answer scoring

TruLens, the RAG triad and answer scoring

TruLens is a Python library for tracing and scoring applications built on language models, with particular emphasis on systems answering questions from documents. Its best known contribution is the RAG triad: three measures that together cover practically every way such an application can fail.

The late July 2026 release carries version 2.10, the licence is MIT, and the required Python starts at 3.10. That distinction deserves noting, since several competing tools in this category ship under licences restricting commercial redistribution, and here no such problem exists.

Who is behind it

The library came out of Truera, a company Snowflake acquired in May 2024. Development continues, and integration with that vendor's ecosystem shows in the documentation and examples.

For a user that carries one practical consequence: plenty of material shows TruLens alongside Snowflake services, which can mislead. The library itself is independent, installs through an ordinary package manager, runs locally, and requires an account with nobody. The judging model can be any provider or a model running on your own hardware.

The RAG triad, this tool's core idea

This is the idea worth learning the library for, even if you end up using another one.

An application answering questions from documents holds three connections: question to retrieved chunks, retrieved chunks to answer, and question to answer. Each can break separately, and each breaks differently.

Context relevance measures the first: whether the retrieved chunks concern the question at all. A low score means the problem sits in retrieval, and no prompt change will fix it.

Groundedness measures the second: whether every claim in the answer can be traced to the retrieved chunks. A low score alongside good context means the model is adding things the documents never contained.

Answer relevance measures the third: whether the answer addresses the question asked. A low score alongside two good ones means the model answered something else, usually a fragment of the question or a neighbouring topic.

The arrangement's strength is that a combination of three scores points at a cause rather than a symptom. A report saying "the system gave wrong information" leads to guessing without these measures. With them you know whether to look in the index, the prompt, or the model.

Installation and first run

pip install trulens trulens-providers-openai

from trulens.core import TruSession, Feedback
from trulens.providers.openai import OpenAI
import numpy as np

session = TruSession()
provider = OpenAI(model_engine="gpt-4o-mini")

context_relevance = (
    Feedback(provider.context_relevance_with_cot_reasons, name="context relevance")
    .on_input()
    .on(context)
    .aggregate(np.mean)
)

groundedness = (
    Feedback(provider.groundedness_measure_with_cot_reasons, name="groundedness")
    .on(context.collect())
    .on_output()
)

The way you declare things here differs from most tools and takes a moment to get used to. Rather than writing a scoring function, you compose one: pick a ready measure, state what it runs on, and define how to aggregate results when several chunks exist.

That last part misleads people. Context relevance is computed separately for each retrieved chunk, so five chunks give five scores. The mean says whether retrieval works generally; the maximum says whether at least one relevant chunk turned up. Those are two different questions and it pays to know which you are answering.

Reasoned variants and what they are for

Measures come in two forms: one returning a number alone, another returning a number together with an explanation.

The reasoned variant costs more tokens and runs slower, and is still usually worth using during development. The reason is simple: a score of 0.4 with no explanation says nothing. A score of 0.4 with a justification stating that two of five chunks concerned a different product points directly at what to fix.

Without justifications you land in the classic trap of working with evaluators: a chart dropped and you cannot tell whether the application got worse or something changed in how scoring works. A justification settles that in a minute.

For runs across a large sample, invert it: the unreasoned variant across the whole set to establish a trend, the reasoned variant on a few dozen worst cases to understand the cause.

The judging model and its trustworthiness

All these measures work the same way: a prompt to a model that issues a score. So the question that fits every such tool needs asking.

The answer is: verify on your own data. Take a hundred cases, label them by hand, compare against the measure, and compute the agreement. Without that number a quality chart shows something you do not know, and decisions taken from it are arbitrary.

The choice of judging model matters more here than assumed. A cheaper, faster model suffices for groundedness, since that task reduces to comparing claims against text. Answer relevance is harder, and a weaker model errs more often there, particularly on multi part questions.

The third thing is language. The measures are described by English prompts and can perform worse on text in other languages, especially for groundedness, where exact matching of a claim to a source sentence matters. The prompts can be changed, and on non English documents that is usually worthwhile.

Beyond the triad, the rest of the measure set

The three main measures cover the most common failures, while the library carries many more, and some of them address problems teams remember too late.

Harmful content detection and sentiment scoring help where answers go straight to a customer. That is not a safeguard in the sense of a real time filter, only a measure telling you how often such output arises at all.

Adherence to a stated style or topic scope is often the most undervalued measure in company assistants. An answer that is factually correct and written in the wrong tone is a real problem, and the triad will not catch it, since on facts everything checks out.

Measures unrelated to the model deserve computing separately. Response time, the number of retrieved chunks, answer length, and token cost need no judge at all, and often explain more than quality scores do. An application whose answers doubled in length after a prompt change usually got worse in reception, even where quality measures showed nothing.

Adding your own measure is simple here, since a feedback function is an ordinary Python function returning a number. In a domain with hard rules, a requirement to cite a legal basis number for instance, a regular expression check is cheaper and more reliable than any judging model.

The cost of scoring and how to limit it

Evaluation can cost more than the application itself, and that surprises teams who priced only the cost of answering users.

The bill follows from arithmetic. The triad's three measures are three extra model calls per answer, and context relevance computed per chunk across five chunks gives seven calls rather than three. Reasoned variants additionally consume output tokens, of which there are plenty, since the model writes an explanation.

Three things reduce that cost without losing value. The first is sampling: scoring every twentieth production conversation gives the same trend as scoring all of them, at a twentieth of the bill. Run the full test set on changes; sample production traffic.

The second is matching the judging model to the measure. Groundedness can be measured with a cheaper model, since the task is comparative. Answer relevance on complex questions deserves a stronger one.

The third is dropping justifications outside diagnostic work. A trend computes from numbers alone, and explanations are needed only when descending into individual cases.

At volume a fourth route makes sense, one that changes the billing unit: run the judging model yourself. A server such as vLLM exposes an OpenAI compatible interface, so you point the scoring provider at your own address and the bill stops depending on how many conversations were scored and starts depending on GPU hours. The break even point sits where scoring runs continuously rather than once per release. The caveat is the one above: agreement between a smaller model and manual scoring has to be measured separately, because a result from one judge does not carry over to another.

Caching deserves a thought too. The same test set run several times a day against an unchanged application need not be recomputed when input and output did not change.

TruLens against the alternatives

The difference between the first row and the rest is one of proportion. TruLens emphasises the measures and their correctness, while the layer for viewing results is thinner. The other tools carry richer consoles and a less refined set of ready measures for document search.

Note that these are not exclusive choices. TruLens can run as a library computing scores in a test pipeline, with results sent wherever the team already looks. Under the MIT licence nothing stands in the way.

Fitting it into a process

The tool delivers most when it runs automatically rather than by hand when somebody remembers.

Start with a test set. Thirty questions with expected answers suffices to begin and beats three hundred collected over a year, since it exists now. The best source is questions users actually asked where the system answered badly.

Then wire triad computation into a pipeline that runs on every change to the prompt, retrieval parameters, or model. Before and after results visible in code review change how a team works more than any console.

Set thresholds per measure. Groundedness should be high and deserves strictness, since it is the measure catching fabrication. Context relevance runs lower and that is normal, since retrieval also returns partially matching chunks.

The last item is watching production traffic. Measures computed on a sample of real conversations once a week show whether the test set still reflects what people ask. It usually does not, and that is precisely the signal to extend it.

Common mistakes

The first is computing one measure instead of three. An answer score alone does not say whether the problem sits in retrieval or in the model, so fixing proceeds blind.

The second is trusting scores without checking agreement with people. A model based measure can err systematically, and without a comparison on manually labelled cases you do not know how often.

The third is using the same model for generating and judging. A model rates its own output more leniently, so results come out higher than they should.

The fourth is the wrong aggregation for context relevance. A mean across five chunks answers a different question from the maximum, and mixing them leads to wrong conclusions.

The fifth is leaving the judging prompts in English while working on documents in another language. Groundedness requires precise claim matching, and that is where the gap is largest.

The sixth is running evaluations once a quarter. A measure not computed on every change does not protect against regression, it documents it afterwards.

The seventh is scoring every production conversation instead of a sample. The trend shows on every twentieth, and the scoring bill can exceed the cost of answering users in the first place.

FAQ

What is the RAG triad?

Three measures covering the three connections in an application answering from documents: relevance of retrieved chunks to the question, groundedness of the answer in those chunks, and relevance of the answer to the question. Their combination points at whether to fix retrieval, the prompt, or the model.

Is TruLens free?

Yes, the library ships under the MIT licence, with no fees and no commercial restrictions. The cost is judging model calls, since every measure is a prompt to pay a provider for or to compute on your own hardware.

Do I need a Snowflake account?

No. The library came from a company that vendor acquired and integrates with its services, while installing through an ordinary package manager, running locally, and accepting any judging model provider.

How does it differ from Phoenix and Langfuse?

In emphasis and licence. TruLens focuses on quality measures, particularly for document search, and ships under MIT. Arize Phoenix offers stronger tracing under the Elastic licence, and Langfuse leans on the choice between self hosting and a cloud.

Can I score applications outside LangChain?

Yes. Ready integrations cover popular libraries, LangChain among them, but the measures compute over any input and output, so your own code calling a model works too.

Documentation sits on the project site, and the code and releases in the GitHub repository.