LangSmith, the end of guessing what the model did

LangSmith, the end of guessing what the model did

An application with a language model breaks differently from ordinary code. There is no exception and no error status, there is an answer that looks right and is wrong. Without a record of what went to the model and what came back, diagnosis means reproducing the situation blind.

LangSmith records that run: every model call, every tool call, tokens spent, timing, and outcome. On top of that it adds an evaluation layer, meaning a way to check whether a prompt change improved behaviour or merely moved the problem elsewhere.

Traces

A trace is a tree of what happened while handling one request. You see each step, its arguments, and its results, so the question "why did the agent answer like that" stops being a mystery.

pip install langsmith
export LANGSMITH_TRACING=true
export LANGSMITH_API_KEY=...

Working with LangChain or LangGraph needs nothing further, since traces are collected automatically. In your own code you mark functions with a decorator.

from langsmith import traceable

@traceable
def answer_question(question: str, user_id: str) -> str:
    fragments = search(question)
    return call_model(question, fragments)

Nested calls to functions marked the same way form a tree, so you see how long retrieval took against generation itself. That settles a common argument about where the latency sits.

Add metadata straight away, since without it a trace is only a record rather than data to analyse. User identifier, prompt version, model name, and request type let you filter and compare later.

@traceable(metadata={"prompt_version": "v3", "type": "support"})
def answer_question(question: str) -> str:
    ...

Datasets and evaluations

This is the tool's other half and the one most often skipped. A prompt change that subjectively improves one answer usually breaks three others, and without a case set nobody notices.

A dataset is a set of examples with expected results. The simplest way to build one is from traces: situations where the application failed get added to the dataset with one click, and from then on every change is checked against them.

from langsmith import evaluate

def correctness(output: dict, expected: dict) -> bool:
    return output["category"] == expected["category"]

results = evaluate(
    lambda inputs: classify(inputs["content"]),
    data="test-tickets",
    evaluators=[correctness],
    experiment_prefix="prompt-v4",
)

Evaluations come in three kinds worth distinguishing. Deterministic evaluation compares a result against an expectation and is cheap and fast. Model as judge evaluation checks things no direct comparison covers, such as tone or factual grounding, and costs tokens. Human evaluation is the costliest and the most reliable, so it is reserved for samples.

Practical advice: start with thirty cases and deterministic evaluation. That suffices to catch a regression, and expanding the set makes sense only once it starts letting errors through.

Costs and latency

Traces carry token counts and timings, so the tool shows what handling one request actually costs. That number often surprises, particularly with agents.

Three things become visible only after enabling traces. The first is retries: an agent correcting an off schema response twice consumes three times what a single call suggests. The second is context growth, where conversation history swells and by the hundredth message input costs more than output. The third is unnecessary calls, retrieval fired on questions that never needed it, for instance.

Filtering by metadata answers questions that would otherwise need your own analytics: which request type costs most, whether a new prompt version shortened response time, which users generate the heaviest usage.

Model as judge evaluation

This is the most used and most misused part of the evaluation layer. The idea is that one model scores another's answer against a stated criterion.

from langsmith import evaluate
from langsmith.evaluation import LangChainStringEvaluator

groundedness = LangChainStringEvaluator(
    "labeled_score_string",
    config={
        "criteria": {
            "grounding": "Does the answer rest solely on the supplied fragments?"
        },
        "normalize_by": 10,
    },
)

Three rules separate useful evaluation from expensive noise. The first is one criterion per judge: asking "is the answer good" yields results incomparable across runs, while asking "does the answer rest solely on the supplied fragments" does not.

The second is a binary or three point scale rather than a ten point score. A model does not distinguish seven from eight repeatably, so the precision is illusory.

The third is checking the judge against a sample scored by a human. If the judge agrees with a person seventy percent of the time, its results still carry information, but a two percentage point difference between prompt versions means nothing.

For applications where factual correctness matters, comparison against a reference answer beats free scoring. It requires preparing references, which is work, and it yields a result a decision can rest on.

Collecting user feedback

Automated evaluation measures what you can describe, while a user sees things you did not anticipate. The tool lets you attach a score to a specific trace from within the application.

from langsmith import Client

client = Client()

client.create_feedback(
    run_id=trace_id,
    key="helpful",
    score=1,
    comment="User clicked thumbs up",
)

The trace identifier must reach the interface alongside the answer so it can later be tied to a click. That implementation detail decides whether feedback can be collected at all.

The value here lies not in a satisfaction statistic but in negatively marked traces forming a natural review queue. From that queue comes a dataset, and from the dataset the next fix. It closes the loop between production and development.

Prompts and versioning

The tool stores prompts as objects with change history, so they can be edited outside the code and their effect on results reviewed.

That carries an advantage and a trap. The advantage is letting a nontechnical person improve a prompt without deploying a new application version. The trap is losing the link between code and prompt: the application fetches the prompt externally, so reproducing last week's behaviour means checking which version applied then.

A sensible compromise pins prompt versions in production code and leaves free editing to the test environment. Deploying a new prompt then becomes a deliberate decision rather than a side effect of somebody else's edit.

Pricing

Beyond the subscription, trace collection itself is billed, and retention is the key factor. A trace kept briefly costs markedly less than one kept long, and the tool automatically extends retention for traces somebody has scored.

The billing scheme was recently reworked towards normalised units, so check the current price list at source before budgeting rather than relying on rates quoted in articles.

A practical saving comes from sampling. In production you rarely need a trace of every request: ten percent of traffic plus every failed case gives the full picture at a fraction of the cost.

Tests in a continuous integration pipeline

Evaluations can run not only by hand but as part of the build, exactly like ordinary tests. That turns answer quality from something checked occasionally into something checked on every change.

import pytest
from langsmith import testing as t

@pytest.mark.langsmith
def test_urgent_ticket_classification():
    text = "I cannot log in since yesterday, urgent"
    t.log_inputs({"content": text})

    result = classify(text)
    t.log_outputs(result)

    assert result["priority"] >= 4

Results land in the tool, so comparison between branches is available in the interface rather than only in build logs. On a prompt change you see immediately which cases stopped passing.

A practical note concerns cost and time. A full set of a hundred cases on every commit means slow tests and a noticeable bill, so a sensible split runs ten fast cases on every change and the full set once a day or before a release.

The second note concerns nondeterminism. A test on a model can pass four times and fail the fifth, so set the pass threshold at set level, requiring eighty percent of cases to pass for instance, rather than at individual call level.

LangSmith against the alternatives

The last row is unfairly overlooked. For an application with one model call, writing prompt, response, and token usage into your own table takes half an hour and suffices. A dedicated tool starts paying off once a call tree appears or version comparison is needed.

The boundary runs roughly where a single call ends and a sequence begins. At three steps with tool calls, your own logging turns into writing your own observability tool, which is work nobody planned for.

One more approach falls outside that table: collecting data through a proxy, meaning you swap the model client's base address instead of adding decorators in code. That is how Helicone works, and on an inherited project whose code you would rather not touch it is the fastest route to cost figures. It carries two limits. A proxy sees single calls rather than a tree of steps, so with an agent it stops helping exactly where it starts being needed. On top of that the product itself moved into maintenance mode after the Mintlify acquisition in March 2026, so it will get no new features and is a poor thing to tie a new project to.

Sensitive data in traces

A trace holds exactly what the user typed and what the model answered. In a customer support assistant that means names, addresses, order numbers, and sometimes data nobody expected.

Three mechanisms handle this. The first is masking on the application side before the trace is sent: patterns detecting card numbers, addresses, and identifiers get replaced with markers. That is the surest option, since the data never leaves your environment.

The second is limiting what reaches a trace at all. Not every call needs to record full content, and the call structure with metadata alone suffices to diagnose most latency and cost problems.

The third is self hosted deployment, available on the plan for large organisations. The question of data transfer then stops existing, at the price of cost and a maintenance obligation.

Settle this at the start rather than after the first audit. Traces collected without masking stay in the system, and removing them retroactively is harder than setting the rules up front.

Common mistakes

The first is traces without metadata. A call record with no prompt version, model, or request type is interesting individually and useless in analysis.

The second is collecting traces without a dataset. You then see that something works badly but cannot tell whether a fix helped until more production data accumulates.

The third is judge model evaluation where a value comparison suffices. A judge costs tokens and adds noise of its own, so reserve it for things no simpler check covers.

The fourth is collecting a hundred percent of production traffic. At high volume the observability bill can exceed the model bill, and sampling gives the same picture.

The fifth is storing conversation content without checking requirements. Traces contain user supplied data, so sensitive information needs masking or a self hosted deployment.

The sixth is freely editing production prompts. Without pinned versions the application's behaviour changes without a deployment, and reconstructing last week's state becomes an investigation.

FAQ

Does LangSmith work only with LangChain?

No. Integration with LangChain and LangGraph is automatic, but your own code takes a decorator and works the same. You can use it with any model vendor and with no agent library at all.

What does LangSmith cost?

The single person plan is free and covers a few thousand traces monthly. The team plan costs thirty nine dollars per seat, up to ten people. Trace collection is billed beyond that, at a lower rate for short retention. The billing scheme changed recently, so check the current price list.

LangSmith or Langfuse?

Choose Langfuse when you need open source and self hosting without negotiation. Choose LangSmith when you work in this ecosystem and want traces and evaluations in one tool rather than assembled from parts.

Can it be self hosted?

Yes, on the plan for large organisations. Under a requirement that conversation content never leaves the company, and with no budget for that plan, an open source tool is the right choice.

Where should I start?

By enabling traces in a test environment and reviewing ten real runs. Two things usually emerge: the step consuming the most time and a call nobody expected. Only then build a dataset from the cases that failed.

Documentation sits on the project site, and current price tiers on the pricing page.