Weave from W&B, tracing and evaluating AI applications

Weave from W&B, tracing and evaluating AI applications

Weave is an observability layer for applications built on language models, made by Weights & Biases. It records the course of every call, runs evaluations against test sets, and compares results between versions.

Its practical value shows on a question hard to answer without tooling: did the prompt change you made on Monday improve the application, or just move the problem. Without a stored test set and before and after results, that is a matter of opinion.

Two things under one name

Before hunting for documentation, separate two products from the same company, since conflating them is the most common source of confusion.

The older product tracks model training: training runs, loss curves, hyperparameter comparisons, dataset and weight versioning. That is a tool for people training models.

Weave is newer and concerns something else: applications that call models. It traces calls, prompts, responses, tool invocations, and costs, and adds evaluations on top. That is a tool for people building products on existing models.

The distinction matters practically, since the packages, documentation, and concepts are separate. A guide describing training runs will not help with tracing an agent, even though both carry similar names and live in the same console.

A change of owner

Weights & Biases is no longer an independent company. CoreWeave, a provider of compute infrastructure for artificial intelligence, announced the acquisition in March 2025 and completed it on 5 May of the same year.

For a user, what mainly changes is the context in which the product develops. An observability tool became part of a compute vendor's offering, alongside credits for runtime environments and inference billed per token.

That cuts both ways. A team already using that infrastructure gets coherent billing and integrations. A team wanting observability alone, with no intention of buying compute, binds itself to a vendor whose main business is something else.

Keep that in mind when planning for years. The tool's direction will from now on follow what helps sell compute, and not necessarily what somebody building an application on other people's models needs.

Instrumentation through a decorator

The way you attach it is the simplest in this whole tool category, and that is a genuine advantage.

import weave

weave.init("support-assistant")

@weave.op()
def retrieve_documents(question: str) -> list[str]:
    return store.search(question, k=5)

@weave.op()
def answer(question: str) -> str:
    chunks = retrieve_documents(question)
    return model.call(question, chunks)

A decorator on a function suffices for its input, output, timing, and errors to reach the console. Nested calls arrange into a tree, so you see that an answer followed from these chunks rather than others.

Calls to popular model providers get captured automatically once initialisation runs, so token usage and cost appear without adding anything.

The simplicity carries a cost worth knowing. The recording format is proprietary, so changing tools means walking through every decorated site. On a short project that means nothing; on a system maintained for years it deserves computing.

There is a second route that older write ups skip, though: Weave also accepts traces in OpenTelemetry format. On multi tenant cloud you POST them to https://trace.wandb.ai/otel/v1/traces, authenticate with the wandb-api-key header, and route them with the wandb.entity and wandb.project resource attributes. The instrumentation can come from OpenInference or OpenLLMetry, and traces can travel through an OpenTelemetry collector. If portability matters to you, instrument with the standard and treat the decorator as a convenient shortcut rather than the only entrance.

Evaluations and scorers

The tool's second half concerns judgement rather than recording.

import weave
from weave import Evaluation

dataset = [
    {"question": "What is the delivery time?", "expected": "3 business days"},
    {"question": "Can I return an item?", "expected": "14 days"},
]

@weave.op()
def score_accuracy(expected: str, output: str) -> dict:
    return {"correct": expected.lower() in output.lower()}

evaluation = Evaluation(dataset=dataset, scorers=[score_accuracy])
await evaluation.evaluate(answer)

A scorer is an ordinary function returning a dictionary, so it can do anything: exact comparison, fuzzy matching, format checking, a call to a judging model. That last variant is the most common with free form answers and the least trustworthy without verification.

Here comes the question to ask of every tool in this class: if a model does the judging, who judges the judge. The answer is always the same and always skipped. Take a hundred cases, label them by hand, compare against the scorer, and compute the agreement. Without that number, a quality chart charts something you do not know.

Evaluation results are versioned, so comparing two runs shows not only the average but which specific cases improved and which regressed. That second column matters more, since an average that rises while an important edge case breaks is a change for the worse.

Note that this same pair, a set of cases and a quality measure, is what the prompt optimiser in DSPy takes as input. There you do not edit the instruction by hand: you describe what goes in and what comes out, and the library picks the prompt text against your measure. Go that way and evaluation stops being a report read after the fact and becomes an objective function, at which point agreement between scorer and human labels counts double. An error in the measure no longer merely distorts a chart; it steers the optimisation the wrong way.

Ingestion based billing

The pricing model separates this tool from its competitors and deserves understanding before production traffic starts.

Rather than counting traces or calls, billing follows the volume of data sent. The consequence is direct: cost depends on how verbose your traces are rather than how many there are.

For a typical question answering application with a short prompt the difference is small. For an application searching documents it is fundamental, since every call drags several text chunks behind it, and those chunks are often longer than the question and answer combined.

The practical conclusion: in such an application, record identifiers and excerpts of retrieved chunks rather than their full text, unless you need the full text for diagnosis. The billing difference is often several fold while the diagnostic value drops little, since a chunk can be fetched back from the store by identifier.

The vendor's price list states this plainly, and three numbers are worth knowing before production traffic starts.

The overage rate looks harmless until you convert it to gigabytes: a hundred dollars per gigabyte above the allowance. An application recording full chunks on every query passes a gigabyte and a half faster than intuition suggests, so this is the line to estimate before rollout rather than after the first invoice.

Cost retention separately. Traces holding full content grow linearly, and with no deletion policy the storage line after a year can exceed the collection line.

Class based scorers and compound judgements

A plain function suffices for unambiguous comparisons; judgements needing state or configuration call for the class based variant.

The difference is practical. A scorer checking a pattern match needs nothing beyond input and output. A scorer calling a judging model needs a client, a judging prompt, a threshold, and usually retry handling for network errors. Keeping that in a function ends in global variables.

For model based judgements three things deserve care. The judging prompt must force a fixed response structure, otherwise parsing the result becomes its own source of errors. The judging model should be cheaper than the one being judged, since you run it across the whole set on every change. And the judgement should return a justification rather than a number alone, since without one you cannot tell whether the scorer erred or the application did.

Multi aspect scoring is a separate matter. One number describing answer quality usually hides more than it shows, since an answer can be accurate and too long, or correct and written in the wrong tone. Several separate scorers, each measuring one thing, produce a picture you can decide from.

What to do with the results

Collecting data is easy; turning it into changes takes a routine the tool will not impose.

A sensible rhythm looks like this. An evaluation running automatically on every change to the system prompt or retrieval parameters, with the result visible before changes merge. That is the only moment where a regression can be stopped more cheaply than after deployment.

The second routine concerns cases that broke. Comparing two runs shows them directly, and reviewing them takes fifteen minutes and says more than an average. A change improving ninety cases and breaking ten is either good or bad depending on which ten.

The third concerns production traffic. Production traces are a source of cases nobody on the team would have imagined, so once a week it pays to review the worst scored ones and move a few into the test set.

Record the rating users give as well, if the interface collects one. A thumbs down tied to a trace gives a label no scorer replaces, and after a few hundred cases lets you check how well automatic scores match what people felt.

Weave against the alternatives

The main argument for the first row is organisational. A team already training models on this platform, holding runs, datasets, and permissions there, gets application observability in the same place, under the same billing and the same accounts.

The argument against comes down mainly to one thing: the data sits with the vendor, which under requirements forbidding prompt text from leaving is decisive and pushes the choice toward locally run options. The complaint about a closed instrumentation format lost its edge once the OpenTelemetry entrance arrived, since instrumenting with the standard moves the trace receiver into configuration rather than into code.

Deploying it sensibly

A few things save time if the tool is to stay in a project.

Start with one function. A decorator on the route handling a query shows within minutes whether the recorded data is what you need. Instrumenting the whole application before checking that is work done blind.

Decide what enters the record. Prompts and responses usually carry user data, so traces become another set governed by the same rules as production data. Masking sensitive data before sending is cheaper than explaining it afterwards.

Split environments into separate projects. Development traces mixed with production ones ruin every statistic and raise the data bill for material nobody looks at.

Build a test set from real cases. The cheapest source is conversations that went badly: every fixed case belongs in the set. After a quarter you hold a set reflecting reality rather than the team's assumptions.

The last item is rhythm. An evaluation run on every prompt change, with the result visible in code review, changes how a team works more than any chart. An evaluation run once a quarter is a ritual.

Common mistakes

The first is confusing the company's two products. Training run documentation does not concern application tracing, even though both share a name and a console.

The second is recording full chunks in a document search application. Under ingestion based billing that is the largest line, and identifiers usually replace it.

The third is trusting a model based scorer without checking agreement against human labels. A chart built on such a scorer shows something you do not know.

The fourth is instrumenting the model call alone. The largest delays and the most frequent errors sit in retrieval and in tools.

The fifth is having no retention policy. Traces grow linearly and after a year the storage line can exceed the collection cost.

The sixth is sending personal data unmasked. Traces land with an external vendor, so prompt content falls under the same requirements as data in a production database.

The seventh is reducing quality to one number. An average from a single scorer hides cases where an answer is factually right and unacceptable for another reason, and those are exactly the ones users report.

The eighth is running evaluations only after deployment. A result before changes merge lets you revert in one move; a result after deployment means explaining a regression somebody already noticed.

FAQ

How does Weave differ from the company's older product?

In purpose. The older one tracks model training, meaning training runs and hyperparameter comparisons. Weave concerns applications calling existing models: it traces prompts, responses, tool calls, and costs, and runs evaluations.

Is Weights & Biases still an independent company?

No. CoreWeave, a compute infrastructure provider, announced the acquisition in March 2025 and completed it on 5 May of the same year, so the tool now forms part of a compute vendor's offering.

What exactly am I paying for?

Billing rests on the volume of data sent rather than the number of traces, plus a storage charge. The free plan covers a gigabyte of data a month, the Pro plan from sixty dollars gives a gigabyte and a half, and the excess costs ten cents per megabyte. That means verbose traces cost more than numerous ones, and document search applications are the most expensive, provided you record chunk text in full.

Can Weave run on my own infrastructure?

The basic arrangement assumes the vendor's cloud, and self hosted options depend on the plan and deserve confirming directly. If local processing is a hard requirement, Arize Phoenix runs on your own server with no negotiation.

Do I have to use a particular library?

No. Instrumentation rests on a decorator applied to your functions, so it works regardless of how you call the model. Calls to popular providers get captured automatically on top of that.

Documentation sits on the product site, and the code in the GitHub repository.