Langfuse, a view into what your LLM application does

Langfuse, a view into what your LLM application does

An application built on a language model fails differently from an ordinary backend. It throws no exception, it returns an answer that looks right and is wrong. Langfuse records every call along with the prompt, the response, timing, and cost, so the question "why did it come out like that" stops being guesswork.

What exactly it records

The base unit is a trace, meaning the record of one run from user input to answer. Inside a trace sit steps: model calls, vector database queries, tool calls, data transformations.

Every step carries four pieces of information that in practice suffice for diagnosis. The input, meaning the exact prompt text together with its context. The output, meaning the raw response before your processing. Duration, letting you point at the stage slowing everything down. Token count and cost, computed from the model and its rates.

That last one tends to be the most revealing. Teams regularly discover that eighty percent of the bill comes from one step nobody thought about, summarising conversation history on every question for instance.

It helps to see how this differs from ordinary application monitoring. Classic tools measure response time, error codes, and resource usage, all of which look healthy with a language model even when the answer is wrong. Here the unit of observation is content rather than status, and that shift is what makes the tool useful.

Traces group by user session and take your own labels, so the question "what is happening for this particular customer" has an answer rather than a hypothesis.

Getting it in

Instrumentation reduces to a decorator or wrapping the model client.

pip install langfuse

LANGFUSE_PUBLIC_KEY=pk-lf-...
LANGFUSE_SECRET_KEY=sk-lf-...
LANGFUSE_HOST=https://cloud.langfuse.com

from langfuse import observe
from langfuse.openai import openai

@observe()
def answer_question(question: str) -> str:
    chunks = retriever.invoke(question)
    result = openai.chat.completions.create(
        model="gpt-5-mini",
        messages=[{"role": "user", "content": build_prompt(question, chunks)}],
    )
    return result.choices[0].message.content

The decorator creates a trace, and the swapped model client automatically attaches the call to it along with tokens and cost. With LangChain and LlamaIndex this works through ready made integrations, with no code change beyond adding a handler.

Writing happens asynchronously, so it does not lengthen the user's response. Do remember the flush call in short lived processes, serverless functions for instance, otherwise the last traces never make it out.

Prompt versioning

The second feature teams reach for concerns prompts. Keeping them in code means every correction needs a deployment, and whoever owns the wording has to ask a developer to move a comma.

from langfuse import Langfuse

langfuse = Langfuse()
prompt = langfuse.get_prompt("ticket-classification", label="production")
text = prompt.compile(ticket=content)

A prompt carries versions and labels, so switching production to a new version means moving a label, and rolling back is the same operation in reverse. Traces record the version used, so comparing quality before and after does not rest on memory.

When fetching a prompt from an external source, add a safeguard against unavailability. The library caches the last fetched version in memory, but on a cold process start it pays to keep fallback text in the code so a tool outage does not stop the application.

Settle one rule early: the prompt in the tool is the source of truth, and only a reference by name stays in the code. Keeping copies in both places ends in a drift nobody notices until the versions start diverging.

Evaluation, or measuring quality

Traces tell you what happened. Evaluation answers whether it was any good, and without it every discussion about improving quality ends in trading impressions.

A dataset is a list of cases with expected results, stored in the tool and run after every prompt or model change. Thirty real cases suffice to notice a regression.

Scores can be assigned three ways, each with a different purpose. By hand in the interface, which works for the first few dozen cases and builds intuition. Automatically through a function comparing output against an expectation, which works for tasks with an unambiguous answer. By a judging model, when correctness is a matter of degree, with summaries for instance.

Collect user scores separately. A thumbs up and down beside an answer costs an hour of work and yields a signal from real usage that no test set replaces.

Pricing and self hosting

The project is MIT licensed, and in June 2025 every product feature, tracing, prompts, and evaluations included, moved under that licence. The self hosted edition has no event or user limits.

Two things set this pricing apart. Plans include unlimited users, so cost does not grow with team size. The free plan, on the other hand, carries a hard cap with no overage billing, meaning tracing simply stops once exhausted rather than generating a bill.

The billing unit counts events rather than model calls, so a trace with five steps consumes more than a single call does. When sizing a plan, compute this against a real run of your own application, since the gap between a simple query and an agent with tools can be tenfold.

Self hosting is a genuine option but not a free one. On top of the machine, add the event database, which grows substantially at volume, plus time for upgrades and backups. With one project and moderate traffic the cloud comes out cheaper, and with data that cannot leave the company the choice is settled regardless of the bill.

How to read traces when diagnosing

Collected data is worth exactly as much as your ability to read it. Three questions reach the cause faster than scrolling through everything.

First: did the model receive the right context. Open the chunk fetching step and read what actually entered the prompt. If the needed information is absent, the problem sits in retrieval and no prompt change will fix it. That is the most common cause of wrong answers in document based systems and simultaneously the one people look for least.

Second: did the model ignore the context it received. If the right chunk was in the prompt and the answer skips or contradicts it, the problem sits in the instruction. Here an order to answer strictly from the supplied chunks and to admit missing data helps.

Third: where did the time go. A trace shows the duration of every step, so the bottleneck is visible immediately. The surprise is often that it is not the model but a database query or an external API call waiting for a response.

With agents a fourth question arrives: how many turns did the loop take. A trace with twenty model calls for one user question usually signals the agent circling rather than a genuinely hard task. A turn limit plus a read of that trace resolves most cases.

Build the habit of opening traces on every user report instead of reproducing the problem locally. Reproduction takes half an hour and often fails, while a production trace shows exactly what happened.

Langfuse against the alternatives

That last row deserves attention, since it gets undervalued. Writing prompts and responses into your own database is an hour of work. The tool starts paying off once you need to compare prompt versions, attribute cost per user, and read multi step traces.

Outside the table sits one more category, tools working as a proxy: instead of decorators you swap the model client base address and the records appear on their own, without touching application code. Helicone works that way, except that after Mintlify acquired it in March 2026 the product sits in maintenance mode and will receive no new features, so for a new project only the Apache 2.0 edition you run yourself is worth weighing.

Cost control in practice

A model bill grows quietly and usually surfaces on the invoice. Traces with computed cost turn that into a number you can watch.

Start by splitting cost across steps. A trace shows what retrieval cost, what answer generation cost, what any history summarisation cost. In most applications one step accounts for the lion's share of the bill, and it is usually a different step from the one intuition suggests.

Then compute cost per user and per session. Labels on traces let you group data, so answering "how many users generate half the bill" takes a minute. That number often matters when setting product pricing, since it shows where the real cost of service sits.

The third step is watching change over time. A bill growing faster than user count means something got longer: conversation history, the context pulled into the prompt, or the number of agent turns. Without a record of the earlier state, that change stays invisible until it becomes painful.

Separately, set an alert on unusual runs. A trace costing ten times the median usually means a loop, an exceptionally long document, or a user probing the system's limits. Each of those deserves seeing immediately rather than a month later.

Finally, record cost alongside a quality score. Without pairing them, cost optimisation often becomes a quality regression nobody notices, because only one side of the equation is measured.

Common mistakes

The first is instrumenting only model calls. The largest value comes from recording the whole run, because in retrieval based systems the fault usually sits in the chunk fetching step rather than in the model.

The second is recording personal data without thinking. Traces hold full prompts, and those hold whatever the user typed. Decide what you mask before the tool reaches production.

The third is leaving retention unset. History grows with traffic, and under self hosting that translates directly into database size and its running cost.

The fourth is collecting traces with no test set. Visibility into what happens does not answer whether a prompt change improved quality. Those are two layers and both are needed.

The fifth is skipping the flush in short lived environments. A serverless function ends before the trace makes it out, so gaps appear in the data that nobody can explain.

FAQ

Is Langfuse free?

The self hosted edition is free under the MIT licence with no event or user limits, you pay only for infrastructure. The cloud has a free plan with fifty thousand units a month, and the Core plan costs 29 USD for a hundred thousand units and unlimited team members.

Langfuse or LangSmith?

Choose Langfuse when you want a local edition, an open licence, or billing independent of team size. LangSmith has deeper integration with the LangChain ecosystem and a more mature interface, so it fits projects built entirely on that framework.

Does tracing slow the application down?

Not noticeably, since writing happens asynchronously in the background. The exception is short lived environments where the process ends before data is sent, which is why serverless functions need an explicit flush before exit.

Does it work with models other than OpenAI?

Yes, it supports models from various vendors including Claude and local models. Cost is computed from the model and its rates, so with a locally served model you see token usage and set the pricing yourself or skip it.

Where do I start with an existing application?

With instrumenting one path, ideally the one causing the most trouble. After a week of collected traces you usually see where the cost sits and where wrong answers originate, and only then is it worth widening the scope and adding a test set.

Documentation sits at langfuse.com, and the source code in the GitHub repository.