Braintrust, an open SDK and a closed scoring platform

Braintrust, an open SDK and a closed scoring platform

Braintrust is a hosted platform for measuring the quality of language model applications: test datasets, scoring functions, call traces, experiment comparisons and a playground for iterating on prompts. The SDK is open source, the platform itself is not, and the bill is driven by how many scores you record and how much data you send. That asymmetry decides when the tool is worth it.

What Braintrust actually does

Four separate things hide behind the name, and mixing them up is the most common source of confusion on first contact.

The first is a library for running offline evaluations. You define a set of cases, a task function and a list of scorers, and the SDK pushes the data through all of it and computes results. The second is a production logger, meaning a record of real model calls with spans, cost and latency. The third is the service itself: the experiment store, the trace browser, charts, a side by side comparison of two runs, and a playground where you change a prompt and immediately see the result on the same dataset. The fourth is autoevals, a separate library of ready made scoring functions.

The distinction matters because parts one, two and four are open and readable, while part three is closed and available only as a service. The SDK can compute results locally, but there is nothing to substitute when you want to see experiment history. Without an account you are left with whatever the console prints.

The integration layer is wide. The npm package exports wrapOpenAI, wrapAnthropic, wrapMistral, wrapCohere, wrapGroq, wrapOllama, wrapGoogleGenAI, wrapAISDK and BraintrustMiddleware, plus bridges to LangSmith through wrapLangSmithClient and wrapLangSmithTraceable. On the Python side the equivalents are named wrap_openai, wrap_anthropic, wrap_litellm, wrap_instructor and setup_pydantic_ai.

An open SDK and a closed platform

This project's licence looks different depending on where you check it, so three places need separate inspection.

The registries say MIT. The npm package braintrust at version 3.28.0 carries "license": "MIT" in its package.json, and the PyPI wheel for version 0.34.0 declares License-Expression: MIT in its METADATA file. The same holds for the autoevals package on npm.

The repositories say Apache 2.0. The LICENSE file in braintrust-sdk-javascript and in braintrust-sdk-python contains the full text of the Apache License, Version 2.0, January 2004, not the MIT text. Neither repository has a LICENSE.md. Both licences are permissive and the practical consequences of the mismatch are small, but a dependency scanner will flag a conflict and your legal team will ask which version governs. The files themselves do not answer that question.

The published packages ship no licence file of their own. The npm archive contains a licenses/ directory with third party licences, specifically for the forks of orchestrion-js, import-in-the-middle and require-in-the-middle, plus a NOTICE file describing those forks, but Braintrust's own licence text is not inside. The PyPI wheel declares no License-File field, and the braintrust-0.34.0.dist-info directory holds no licence file at all. The code is real, though: the npm archive weighs about 4.3 MB and the Python wheel contains 374 files, including framework.py, logger.py and oai.py.

The reverse situation in autoevals 0.3.0 is interesting. The PyPI wheel does contain autoevals-0.3.0.dist-info/licenses/LICENSE with MIT text, but its METADATA has neither a License-Expression field nor a licence classifier, which makes the PyPI API return license: null for the package. A scanner reading only metadata will treat the package as unlicensed even though the file sits inside. The autoevals repository has an ordinary LICENSE file with MIT text signed BrainTrust Data, 2023.

The platform itself is not open source and has no self hosted variant on the ordinary plans. The pricing page lists on-premise or hosted deployment only under Enterprise with custom pricing. That is not self hosting in the sense Langfuse means it, it is a contract.

Two version lines, two repositories

The numbering is confusing and that is not your fault. The npm package braintrust is at 3.28.0 while the PyPI package of the same name is at 0.34.0. Both shipped on 17 August 2026, npm at 19:08 UTC and PyPI at 19:29 UTC, roughly half an hour apart.

These are not two branches of the same code, nor a mirror. They are two independent repositories with separate release cycles. The repository field in the npm package points to braintrustdata/braintrust-sdk-javascript with the js subdirectory, and project_urls on PyPI points to braintrustdata/braintrust-sdk-python. The historical address github.com/braintrustdata/braintrust-sdk, still circulating in documentation and blog posts, now answers with a 301 redirect to the JavaScript variant.

The release feeds confirm the split. The JavaScript repository's feed lists entries formatted as braintrust@3.28.0, braintrust@3.27.0 from 4 August and braintrust@3.26.0 from 1 August, plus the separately released @braintrust/otel@0.3.0. The Python repository's feed uses a different format: Python SDK v0.34.0, Python SDK v0.33.0 from 11 August, Python SDK v0.32.0 from 5 August.

The practical consequence is that a version number in a documentation example tells you nothing until you check which language it refers to. I checked the peer dependency ranges separately and there is no trap here: @braintrust/otel 0.3.0 declares braintrust in the range >=1.0.0-0, so it installs alongside core 3.28.0 without conflict.

npm install braintrust autoevals
pip install "braintrust[cli]" autoevals

export BRAINTRUST_API_KEY=your_key
npx braintrust eval tutorial.eval.ts
braintrust eval eval_hello.py

A first eval and production logging

The evaluation shape is the same in both languages and comes down to three arguments: data, task and scores. In Python, Eval additionally accepts experiment_name, trial_count, metadata, tags, max_concurrency, timeout, parameters, error_score_handler and no_send_logs, and asynchronous code has a separate EvalAsync function.

from braintrust import Eval
from autoevals import Factuality, Levenshtein

def task(input, hooks):
    hooks.metadata["prompt_version"] = "v3"
    return "Hi " + input

Eval(
    "Say Hi Bot",
    data=lambda: [
        {"input": "Foo", "expected": "Hi Foo"},
        {"input": "Bar", "expected": "Hello Bar"},
    ],
    task=task,
    scores=[Levenshtein, Factuality],
    trial_count=3,
    max_concurrency=8,
    tags=["regression"],
)

In TypeScript the signature is Eval(name, evaluator, reporterOrOpts?), and the evaluator object's fields are camelCase: data, task, scores, classifiers, experimentName, trialCount, metadata, tags, isPublic, update, maxConcurrency, timeout. Production logging goes through initLogger with the fields projectName, projectId, environment, asyncFlush, apiKey and appUrl.

import { initLogger, wrapOpenAI, wrapTraced, flush } from "braintrust";
import OpenAI from "openai";

initLogger({
  projectName: "customer-support",
  environment: "production",
  asyncFlush: true,
});

const client = wrapOpenAI(new OpenAI());

export const answer = wrapTraced(async function answer(question: string) {
  const res = await client.chat.completions.create({
    model: "gpt-5-mini",
    messages: [{ role: "user", content: question }],
  });
  return res.choices[0].message.content;
});

await flush();

The key is read from the BRAINTRUST_API_KEY variable, and in Node.js additionally from the nearest .env.braintrust file in the current or a parent directory. Diagnostics are switched on with BRAINTRUST_DEBUG_LOG_LEVEL set to error, warn, info or debug. The Python package registers itself as a pytest plugin through the pytest11 entry point, and the JavaScript side offers wrapVitest and a vitest-evals-reporter export, so evaluations can live inside your existing test suite instead of next to it.

Autoevals without a Braintrust account

This is the most important fact for anyone worried about vendor lock-in. The autoevals library is a separate MIT licensed package and works without a Braintrust key. Its init function accepts a client argument, which can be any client compatible with the OpenAI API, plus default_model for choosing the judge model. The default is gpt-5-mini.

The scorers fall into three groups. Deterministic ones call no model and compute locally: Levenshtein, NumericDiff, JSONDiff, ValidJSON, ExactMatch, ListContains. Judge scorers call a model and return a score with a rationale: Factuality, Battle, ClosedQA, Humor, Possible, Security, Sql, Summary, Translation. The third group covers retrieval system metrics, carried over directly from the naming familiar from Ragas: Faithfulness, AnswerRelevancy, AnswerCorrectness, AnswerSimilarity, ContextPrecision, ContextRecall, ContextRelevancy, ContextEntityRecall.

from openai import OpenAI
from autoevals import init, Factuality, Levenshtein

init(client=OpenAI(base_url="http://localhost:11434/v1", api_key="none"),
     default_model="gpt-5-mini")

result = Factuality()(
    input="Who wrote The Doll?",
    output="Boleslaw Prus",
    expected="Boleslaw Prus",
)
print(result.name, result.score, result.metadata)

print(Levenshtein(output="cat", expected="cax").score)

The result object has the fields name, score between zero and one or null, and metadata. The dependencies are worth knowing too, because they differ between languages. The Python version pulls chevron, jsonschema, polyleven and pyyaml, with the openai package present only in the development extra. The JavaScript version has openai in the range ^6.7.0 as a hard production dependency, so it installs even when you use no OpenAI model at all. On top of that it needs zod in the range ^3.25.0 || ^4.0.0 as a peer dependency.

On the exit question: the scorers come with you, the experiment history does not.

Pricing and the billing unit

The pricing page renders without JavaScript, so the numbers below come from raw HTML fetched on 22 August 2026. Billing is hybrid and consists of four lines: a fixed platform fee, model credits, processed data measured in gigabytes, and scores counted individually.

The definitions in the questions section are precise and they matter. A score is every individual recorded result: "each time you record a score, the total number of scores counted towards your monthly usage increases by one". Processed data means data sent to Braintrust, not storage occupied, and once you pass the quota, deleting data does not lower the counter.

Take a team running a thousand scores a day, meaning thirty thousand a month over a thirty day month. On Starter, ten thousand are included and the remaining twenty thousand cost twenty times $2.50, so $50, against a zero platform fee. On Pro, thirty thousand fits inside the fifty thousand allowance, so scores cost nothing, but the platform fee is $249. At that volume Starter is five times cheaper. The break-even point on the score line alone falls at 199,000 scores a month, roughly 6,633 a day: Starter pays $2.50 times 189, Pro pays $249 plus $1.50 times 149, and both come to $472.50. Stated plainly as a caveat: this arithmetic covers the score line only. Processed data cannot be converted without knowing the size of a single trace, so I deliberately do not guess it.

What happens after you exceed the free plan quota is only partly described. The questions section says Starter needs no credit card and that usage beyond the quota is enabled with an "on-demand usage" switch in billing settings, or by moving to Pro. Exactly what stops working if you leave that switch off is not spelled out on the pricing page.

One discrepancy on that same page. The plan card and the feature table give the user count as unlimited on all three plans, while the answer to "Which plan is right for me?" describes Pro as best suited for teams of up to five people. Both numbers come from the same document and they are not reconciled. The startup programme gives 6 to 12 months of Pro for free, provided you are a new customer with at least $100K raised. No annual price is published, so there is no annual arithmetic to check.

Braintrust against Langfuse, LangSmith, Promptfoo and Ragas

This collection already covers four related tools, and the differences between them come down to two axes: what is open, and where the data lives.

Promptfoo is the opposite of Braintrust on entry cost: you write a config file, run it locally and create no account. Langfuse is the choice when data has to stay on your own infrastructure without a sales conversation. LangSmith makes sense when you are standing on LangChain anyway. Ragas is a metrics library rather than a platform and solves a narrower problem.

Braintrust wins in one scenario: when more people than the code author work on the prompt. A playground where a product manager swaps a prompt and sees the result on the same dataset, plus a side by side comparison of two experiments, are things you will not build in half a day. For a one person project where you do everything from the terminal anyway, it is overkill.

Common mistakes

The first is looking for the repository at the old address. braintrustdata/braintrust-sdk is a redirect today and the Python code lives elsewhere.

The second is comparing version numbers across languages. A three on npm and a zero point something on PyPI describe the same product in the same week. When reading changelogs, always check whose feed it is.

The third is assuming that because the package is MIT, the platform is too. The SDK and autoevals are open. The experiment store, the interface and the playground are not.

The fourth is forgetting flush with asynchronous writes. With asyncFlush set to true, a function in a serverless environment can finish before the trace has been sent.

The fifth is costing the service by scores alone. Processed data grows independently, and once past the quota, deleting data changes nothing because the counter is cumulative.

The sixth is relying on retention. Fourteen days on the free plan means a comparison against a three week old experiment will not be possible, and S3 export is an Enterprise feature. If history matters, keep your own copy of the results in the repository alongside it.

FAQ

Is Braintrust open source

Partly. The npm and PyPI SDKs and the autoevals library are open, though the registries declare MIT while the LICENSE files in the repositories contain Apache 2.0 text. The platform, meaning the experiment store, the interface and the playground, is closed and available as a service.

Can autoevals be used without a Braintrust account

Yes. The package is independent, and its init function takes any client compatible with the OpenAI API, including a local one. Deterministic scorers such as Levenshtein or JSONDiff need no model at all.

Why is npm at 3.28.0 and PyPI at 0.34.0

Because these are two separate repositories with independent release cycles, despite the identical package name. The 17 August 2026 releases landed on the same day about half an hour apart, but the numbers were never synchronised.

What does a thousand scores a day cost

On Starter, $50 a month for the scores alone, because ten thousand of the thirty thousand are included and the rest costs $2.50 per thousand. On Pro the scores fit inside the allowance but you pay the $249 platform fee. Processed data comes on top and cannot be estimated without knowing the size of a trace.

Can Braintrust be run on your own infrastructure

Not on Starter or Pro. The pricing page lists on-premise or hosted deployment only under Enterprise with custom pricing, which means a contract rather than pulling an image.

When is something else the better choice

When you work alone from the terminal, Promptfoo is simpler and cheaper. When data cannot leave your infrastructure without negotiation, pick Langfuse. When you evaluate a retrieval system only, Ragas as a library is enough.