Arize Phoenix, tracing and evaluating LLM applications

Arize Phoenix, tracing and evaluating LLM applications

Arize Phoenix records the course of every call in an application built on a language model: the prompt, the answer, the tools invoked, the document chunks retrieved, timing, and tokens consumed. On top of that it adds a set of evaluators that score answer quality without a human in the loop.

The problem it solves appears with the first user report. Somebody writes that the system gave wrong information, and you hold one sentence and no way to reconstruct what happened. Across three steps with tools and document search, guessing stops working.

The licence, the thing to check first

The tool gets described as open source, and that is a simplification worth unpacking before you write it into a product.

The code is public and you can run it yourself at no charge, while the licence is Elastic License 2.0 rather than one approved as open under the accepted definitions. The practical difference comes down to one prohibition: you may not offer this software to others as a managed service.

For most teams that changes nothing. You run it on your own server, observe your own application, end of story. It starts mattering when you are building a platform where your customers would receive an observability console as part of your offering. That is precisely the case the licence rules out.

Settle it at the start, since replacing an observability layer after a year of work is expensive, and a conversation with legal after signing a customer contract can be costlier still.

Installation and first run

pip install arize-phoenix
phoenix serve

You can also run it without installing anything permanently:

uvx arize-phoenix serve

Releases land several times a week, so the version number moves faster than any text can keep up with: in mid August 2026 it stood at 19.21. Python from 3.10 up to 3.14 is required. The interface comes up locally and accepts data immediately, so you see the first traces within minutes of installing.

For a permanent deployment there are container images and charts for running in a cluster, plus one click deployment buttons for several cloud providers. The choice depends on whether the data must stay with you: running locally or in your own infrastructure settles that requirement without negotiation.

Instrumentation through OpenTelemetry

This is the tool's biggest advantage and worth understanding, since it concerns a decision that lasts years.

Data collection rests on OpenTelemetry, the standard used well outside the language model world, extended with a description of concepts specific to this field: prompt, response, tool call, retrieved document.

The consequence is practical. You instrument the application once, and can then swap the tool receiving the traces without touching code, since they speak the same language. That separates this approach from libraries with their own closed format, where changing vendors means walking through every call site.

from phoenix.otel import register
from openinference.instrumentation.langchain import LangChainInstrumentor

register(project_name="assistant")
LangChainInstrumentor().instrument()

Ready instrumentations cover popular libraries, LangChain, LlamaIndex and Haystack among them, plus direct calls to the major model providers. With your own code you add spans by hand, which across well divided functions takes a dozen or so lines.

Of those three the last gives the most readable trace, because the application there is a graph of components with declared inputs and outputs, so the span tree matches the drawing of the pipeline rather than an accidental nesting of functions. With your own code you have to earn that yourself, naming spans so they still read six months later.

Decide up front what enters the traces. By default the full text of prompts and responses gets recorded, so if personal data passes through them, traces become another place needing retention and access control.

Evaluators, or scoring without a human

Tracing says what happened. Evaluation says whether it was any good, and that is the tool's second half.

Built in evaluators cover grounding of answers in the supplied sources, relevance of retrieved chunks, detection of fabricated content, toxicity scoring, and criteria you write yourself. Each of them is in practice a prompt to a judging model, with a defined response format.

The first three overlap with what TruLens calls the RAG triad: relevance of the retrieved chunks, grounding of the answer in them, and relevance of the answer itself. You reach for that library when what counts most is measures for document retrieval plus an MIT licence with no ban on reselling as a service, and the price is a far plainer interface than this one.

That raises the question that always comes up, and rightly: if a model does the judging, who judges the judge. The answer is you, on a set of examples labelled by hand. A hundred cases with manual labels suffices to check how often the evaluator agrees with a person, and that is the only way to know what its results are worth.

Evaluator prompts are visible and editable, which separates this from closed scoring that returns a number with no justification. In a domain departing from the general, legal or medical for instance, that capability is decisive, since a general evaluator will err systematically.

Evaluation runs against a set of stored traces, so a typical cycle looks like this: collect production traffic for a week, pick a few hundred cases, run the evaluators, inspect the worst results. That last step matters most and gets skipped most often, since it is where you see what actually breaks in the application.

Experiments and prompt versioning

The third layer concerns changes you make deliberately.

Datasets are sets of inputs with an expected result, stored and versioned. An experiment means running the application against such a set and comparing results with a previous run.

The value shows when changing a system prompt. Without a test set, changing a prompt is a move in the dark: you fix one case, break three others, and learn about it from reports. With a set you see both numbers before deployment.

Prompts can be stored in the tool itself, with version history, which is convenient, though worth thinking about whether you want that. A prompt is part of application behaviour, so keeping it in the repository alongside code gives change review, a link to the deployment, and single step rollback. A separate place with its own history is more convenient for non technical people and worse for deployment coherence.

What a trace actually shows

Worth knowing what to look for, since a first glance at a call tree can overwhelm, while most answers sit in three places.

The first is the time breakdown. A trace shows how long each step took separately, and that usually demolishes the team's assumption about what is slow. In applications searching documents the model is often the fastest element, with time going to the vector database query, computing the query embedding, and sequential tool calls that could have run in parallel.

The second is the retrieved chunks. On an answer that contradicts the facts, the first question is whether the right chunk reached the context at all. If not, the problem sits in retrieval and no prompt change will fix it. If it did and the model still answered otherwise, the problem sits in the prompt or the model. That distinction turns the work around completely, and without a trace you cannot make it.

The third is tool call arguments. An agent that called the right function with the wrong arguments returns an answer that looks sensible and is entirely wrong. You see it only in the call record, since the answer itself carries no trace of what went to the tool.

The fourth, used less often, is token consumption broken down by step. With an agent taking several turns the total often runs several times higher than the conversation length suggests, since each turn carries the whole context so far. That is where savings are easiest to find.

From traces to fixes

Collecting data is easy; drawing conclusions from it takes a routine nobody will impose for you.

A sensible cycle looks like this. Once a week you take twenty lowest scored conversations, read them, and assign a cause: bad retrieval, bad prompt, a missing tool, a data error. After four weeks you hold a distribution of causes and know where to work, instead of fixing whatever caught your eye.

The second routine concerns changes. Every modification to a system prompt or retrieval parameters should pass through the same test set, with the before and after recorded. Without it a team six months in cannot tell whether things improved since the start, since everyone remembers only the last change.

The third concerns edge cases. A conversation that went badly and got fixed belongs in the test set as a case. That is the cheapest way to build a set reflecting reality, since the examples come from users rather than from the team's imagination.

Record the rating users give as well, if the interface collects one. A thumbs down tied to a trace gives a label no evaluator replaces, and after a few hundred such cases you hold material for checking how well automatic scores match what people felt.

Phoenix against the alternatives

The last row deserves honest consideration and gets skipped. If the team runs a working trace collection and dashboard system, adding a few attributes describing model calls to it is cheaper than introducing another tool with a separate interface and separate upkeep. You lose the ready evaluators and gain one place instead of two.

The difference between the first three rows comes down mainly to where the data sits and which licence binds you. Their tracing capabilities are comparable, and which one leads on a specific feature comparison shifts quarter by quarter.

Deployment in practice

A few things save time if the tool is to stay beyond a week of exploration.

Start locally. Running it on your own machine and pointing the development environment at it takes fifteen minutes and immediately shows whether the traces hold what you need. Only then does a conversation about a permanent deployment make sense.

Set retention early. Traces carrying full prompt text grow faster than assumed, particularly in applications searching documents, where every query drags several text chunks behind it. With no deletion policy the database grows linearly forever.

Separate projects by environment. Development traces mixed with production ones ruin every statistic, since experiments look like user errors.

The last item is habit. An observability tool nobody opens is a cost with no return. One weekly review where somebody inspects the ten worst scored conversations delivers more than the richest set of charts.

Common mistakes

The first is treating the licence as open. Elastic License 2.0 forbids offering this software to others as a managed service, so a plan to embed the console in a customer offering needs checking before work begins.

The second is trusting evaluators without verification. A score issued by a model can err systematically, and without a manually labelled set you do not know how often.

The third is recording full content with no retention policy. Traces then become a copy of production data, carrying the same requirements without the same protection.

The fourth is instrumenting the model call alone. The largest delays and the most frequent errors usually sit in document retrieval and tool calls rather than in the model itself.

The fifth is mixing environments in one project. Statistics stop meaning anything once half the traces come from testing.

The sixth is collecting data with no recurring review. Without somebody regularly inspecting the worst cases, the whole deployment becomes an archive.

FAQ

Is Arize Phoenix free?

To run yourself yes, with no charge and no feature gating. The licence, though, is Elastic License 2.0 rather than an open licence in the accepted sense, and it forbids offering this software to others as a managed service.

Does data leave my infrastructure?

Running locally or in your own container, no, and that is the main reason to pick this tool. A vendor hosted variant also exists, where traces land on their servers.

How does it differ from Langfuse and LangSmith?

Mainly in licence and where the data sits. Langfuse offers a choice between self hosting and the vendor's cloud, LangSmith is a closed service tied to the LangChain ecosystem, and Phoenix leans on self hosting with OpenTelemetry based instrumentation.

Do I have to use LangChain for this to work?

No. Ready instrumentations cover popular libraries, but data collection rests on OpenTelemetry, so your own code calling a model directly can be described too, by adding spans manually.

Can the evaluators be trusted?

Only after verification against your own data. An evaluator is a prompt to a judging model, so its results need comparing with manual labels across a few hundred cases before anybody decides anything from a chart.

Documentation sits on the project site, and releases and code in the GitHub repository.