Opik, tracing and evaluation for LLM applications

Opik, tracing and evaluation for LLM applications

Opik is Comet ML's platform for collecting traces of language model calls and evaluating them. Version 2.2.36 shipped on 21 August 2026 and appeared in PyPI and npm on the same day, both packages under Apache 2.0. The entire codebase lives in one repository under one license, with no directory carrying separate terms, and in this category that is rare.

What Opik does and where it comes from

Opik has three layers, and separating them immediately clarifies how to think about cost.

The first layer is the client libraries: the opik package on PyPI and the opik package on npm. They collect traces, meaning trees of spans that describe one request: a model call, a tool call, an agent step. The second is the server with its databases and dashboard, run either on your own machines or consumed as a hosted service at https://www.comet.com/opik/api. The third is the evaluation library: datasets, scoring functions, experiments and comparisons between them.

Comet ML is an older company than Opik. Its original product is an MLOps platform for tracking model training, and Opik is a separate product line for generative applications. On the pricing page both families have their own plan tables and their own billing units, which is easy to misread on a first pass.

The comet-ml/opik repository answers with a 200 and does not redirect anywhere else. The Atom release feed is harder to read: next to the actual 2.2.36 release of 21 August 2026 it holds dozens of continuous integration build tags such as 2.2.37-6393 and 2.2.37-7950-merge-3036, published the same day. For version tracking the package registries are more reliable here than the release list.

The matching version numbers across registries deserve their own sentence, because in this category that is not the norm. PyPI and npm both carry 2.2.36 for Opik with the same date. By comparison, npm langfuse sits at 3.38.20 from 1 April 2026 while PyPI langfuse sits at 4.14.4 from 11 August 2026. For Braintrust, npm shows 3.28.0 and PyPI shows 0.34.0. If you pin versions in two languages at once, shared numbering saves a lot of bookkeeping.

The Python package also installs an opik console script and registers a pytest plugin through the pytest11 entry point pointing at opik.plugins.pytest.hooks. That explains one of the items you are about to see on the dependency list.

The license checked in three places

I checked the license in three independent sources, because in this category of tooling the registry declaration often disagrees with what the package actually contains.

The repository has a LICENSE file and only that one; LICENSE.md returns 404. It is 203 lines long, starts with the line Copyright (c) Comet ML, Inc, continues with the standard Apache License 2.0 text, and its appendix reads Copyright 2024 Comet ML, Inc. There are no clauses added on top of the boilerplate. Separate, identical copies sit in sdks/python/LICENSE and sdks/typescript/LICENSE.

The license field in the registries looks different depending on the ecosystem. npm reports Apache-2.0, a valid SPDX identifier. PyPI reports the string Apache 2.0 License and its license_expression field is empty. That is free text, not an identifier, so a dependency audit tool matching on SPDX will not recognise it and will need a manual rule. This is still better than pasting the whole license text into that field, which some other projects do.

That leaves the contents of the published packages. The wheel opik-2.2.36-py3-none-any.whl contains the opik/ and _opik/ directories with real code plus a license file at opik-2.2.36.dist-info/licenses/LICENSE, again Apache 2.0. The npm archive opik-2.2.36.tgz contains LICENSE, README.md and a dist/ directory with index.cjs, index.js and type declarations. Neither package is a stub and neither is missing a license file.

That leaves the question that matters most for evaluation platforms: whether the repository hides a server directory under a separate, more restrictive license. I checked the paths where such files usually sit: apps/opik-backend/LICENSE, apps/opik-frontend/LICENSE, apps/opik-guardrails-backend/LICENSE, deployment/helm_chart/opik/LICENSE and NOTICE in the root. All return 404. I state the assumption plainly: I checked candidates, not the whole repository tree, because I did not use the GitHub programmatic interface. With that caveat, the picture is clean, Apache 2.0 across the board, and that is a genuine differentiator against tools that keep the server under a reciprocal license and the client under a permissive one.

Installation, dependencies and a first trace

Installation is short; the dependency list is what deserves a look before the package goes into a production environment.

# Python client, requires Python 3.10 or newer
pip install opik

# interactive setup: writes the key and host into ~/.opik.config
opik configure

# TypeScript client, requires Node 18 or newer
npm install opik

# running the platform yourself
git clone https://github.com/comet-ml/opik.git
cd opik
./opik.sh                      # full stack, dashboard at http://localhost:5173
./opik.sh --infra              # databases and object storage only
./opik.sh --backend            # infrastructure plus backend, no dashboard
./opik.sh --guardrails         # with the guardrails service
./opik.sh --verify             # container health check
./opik.sh --stop
./opik.sh --clean              # wipes the data

On the Python side the interpreter must be at least 3.10. Among the mandatory, not optional, dependencies sit pytest and sentry_sdk>=2.0.0. Pytest is there because the package registers a test plugin, but the effect is that a telemetry library drags a test framework into your production image. Next comes litellm with a long list of exclusions: !=1.81.*, !=1.82.*, !=1.83.0 through !=1.83.6, !=1.92.*, and for Python older than 3.11 an additional upper bound of <1.97 that no longer applies on 3.11 and above. Such a list is a record of broken upstream releases and a signal that upgrading litellm in your project can collide with this constraint. On top of that come boto3-stubs[bedrock-runtime], openai, pydantic, rich, tenacity, uuid6, jinja2, watchfiles and three tree-sitter packages, skipped on Linux aarch64. The proxy extra adds fastapi and uvicorn.

The TypeScript client has an entirely different dependency set and conclusions must not be carried from one to the other. It has twenty six runtime dependencies, including ai at ^6.0.13 and four @ai-sdk provider packages: openai, anthropic, google and google-vertex. Its peer dependency is zod in the range ^3.25.55 || ^4.0.0.

Here is a defect worth preparing for. The integration packages opik-openai and opik-langchain carry version 2.2.36, the same as the core, but their peer dependency ranges were left behind in the previous version line: opik-openai asks for opik in the range ^1.8.61 and opik-langchain in the range ^1.8.75. Neither range covers 2.2.36, so installing the core alongside an integration ends in an ERESOLVE conflict on npm 7 and newer.

Instrumenting the code itself is simple. The @track decorator accepts, among others, name, type, tags, metadata, capture_input, ignore_arguments, capture_output, flush, project_name and environment.

import opik
from opik import Opik

@opik.track(
    name="retrieve_context",
    type="tool",
    tags=["rag", "production"],
    capture_output=False,
    ignore_arguments=["api_key"],
)
def retrieve(query: str, api_key: str) -> list[str]:
    return ["chunk 1", "chunk 2"]

@opik.track(name="answer", type="llm", entrypoint=True)
def answer(question: str) -> str:
    context = retrieve(question, api_key="secret")
    return f"Answer based on {len(context)} chunks"

client = Opik(project_name="assistant", batching=True)
answer("How long is span retention?")
client.flush()

The ignore_arguments parameter matters here for practical reasons, because capture_input defaults to recording every function argument, keys and personal data included. The batching parameter defaults to on, and the docstring of the Opik class warns about it directly: update operations, meaning update_span and update_trace, may lose data if the update reaches the server before the batched create request is flushed.

Datasets and experiments

Evaluation in Opik rests on a dataset, a function that performs the task, and a list of metrics. The evaluate function joins those three into an experiment visible in the dashboard.

import opik
from opik.evaluation import evaluate
from opik.evaluation.metrics import Hallucination, Equals

client = opik.Opik()
dataset = client.get_or_create_dataset(
    name="production-questions",
    description="Traffic sample, August 2026",
)
dataset.insert(
    [
        {"input": "How long is retention?", "expected_output": "60 days"},
        {"input": "Is there RBAC in self-host?", "expected_output": "no"},
    ],
    num_threads=1,
)

def task(item: dict) -> dict:
    return {
        "input": item["input"],
        "output": my_agent(item["input"]),
        "reference": item["expected_output"],
        "context": [],
    }

result = evaluate(
    dataset=dataset,
    task=task,
    scoring_metrics=[Equals(), Hallucination(model="openai/gpt-4.1-mini")],
    experiment_name="prompt-version-7",
    experiment_config={"prompt_version": 7, "temperature": 0.2},
    task_threads=16,
    nb_samples=200,
    trial_count=1,
    experiment_tags=["regression"],
)

A few details of that signature matter in daily work. The task_threads parameter defaults to 16, so without changing it you hit your model provider with sixteen concurrent requests and may run into rate limits. The nb_samples parameter caps the number of items, which is the cheapest way to do a trial run before the full pass. The project_name parameter in evaluate is marked deprecated: if the dataset has project_name set, this argument is ignored and the user gets a warning. The scoring_key_mapping parameter remaps keys when the task returns names other than those the metric's score method expects.

The dataset.insert method creates a new dataset version on every call. Its documentation also says that with num_threads greater than one the batches are uploaded in parallel, and that on failure of one batch the exception is re-raised with no rollback, so batches already uploaded stay on the server.

A separate evaluate_experiment function accepts experiment_name, scoring_metrics and scoring_threads and adds metrics to an existing experiment without re-running the task. That matters for cost: if you added a new judge metric, you do not have to run the whole application again, only the scoring stage.

The TypeScript client has an equivalent of the same shape, with fields in camelCase.

import { evaluate, Opik } from "opik";

const client = new Opik();
const dataset = await client.getOrCreateDataset("production-questions");

const result = await evaluate({
  dataset,
  task: async (item) => ({ output: await myAgent(item.input as string) }),
  scoringMetrics: [],
  experimentName: "prompt-version-7",
  experimentConfig: { promptVersion: 7 },
  nbSamples: 200,
});

Metrics, or what is deterministic and what pays for a judge

This is the most practical part, because the evaluation bill can exceed the bill for the application itself. In package 2.2.36 the opik/evaluation/metrics/heuristics directory holds twenty modules and the llm_judges directory holds thirteen subpackages.

Deterministic metrics compute locally and generate no traffic to a model provider. They include Equals, Contains, RegexMatch, IsJson, LevenshteinRatio, SentenceBLEU, CorpusBLEU, ROUGE, GLEU, ChrF, METEOR, Readability, Tone, PromptInjection, SpearmanRanking, Sentiment, VADERSentiment and the trio JSDivergence, JSDistance and KLDivergence. The Equals.score method takes output and reference, while PromptInjection works off a list of regular expressions and keywords supplied through patterns and keywords.

There is a middle category that is easy to forget. BERTScore takes a model_type that defaults to bert-base-uncased and computes on a transformer model fetched locally, while LanguageAdherenceMetric takes expected_language and an optional model_path to a fastText model. These are not calls to a paid interface, but they are not free arithmetic either: they need memory, time and a weights download.

Some heuristic metrics have dependencies outside the base install. SentenceBLEU and CorpusBLEU require nltk and raise an exception telling you to install it, while Readability uses the textstat module. Neither is a dependency of the opik package, so a first run in a clean environment fails.

Judge metrics call a model for every item in the dataset. They are AnswerRelevance, ContextPrecision, ContextRecall, Hallucination, Moderation, Usefulness, TrajectoryAccuracy, SycEval, StructuredOutputCompliance, LLMJuriesJudge and GEval with a set of ready-made presets, including AgentTaskCompletionJudge, AgentToolCorrectnessJudge, QARelevanceJudge, SummarizationCoherenceJudge, GenderBiasJudge and PoliticalBiasJudge. Conversation metrics form a separate group, for example ConversationalCoherenceMetric, UserFrustrationMetric and SessionCompletenessQuality.

The default judge model is written into the code and is openai/gpt-5-nano, in the default_llm field of the OpikConfig class. The call goes through LiteLLM, so the model name takes the provider-prefixed form. This is worth knowing, because calling Hallucination() with no arguments quietly picks a model that you pay for, not Comet.

from opik.evaluation.metrics import (
    Equals, RegexMatch, IsJson,       # deterministic, no bill
    Hallucination, AnswerRelevance,   # judge, one model call per item
    GEval,
)

cheap = [Equals(), RegexMatch(regex=r"^\d{4}-\d{2}-\d{2}$"), IsJson()]

expensive = [
    Hallucination(model="openai/gpt-4.1-mini", temperature=0.0, seed=42),
    AnswerRelevance(model="openai/gpt-4.1-mini", track=False),
    GEval(
        task_introduction="You judge whether the answer stays within the context.",
        evaluation_criteria="1 means unrelated, 5 means fully grounded.",
        model="openai/gpt-4.1-mini",
        temperature=0.0,
        reasoning_effort="low",
    ),
]

result = Hallucination().score(
    input="How long is retention?",
    output="Sixty days.",
    context=["Span retention on the Free plan is 60 days."],
)
print(result.value, result.reason)

Three details from the code above. First, every metric has a track parameter defaulting to True, meaning the metric result is written as a separate trace. On the hosted service, which bills by spans, evaluation therefore inflates the very counter your bill depends on. Second, GEval accepts reasoning_effort and its documentation states that the provider default applies when it is unset, typically medium, so setting low is a way to cut reasoning tokens. Third, the score method signatures differ: Hallucination.score and AnswerRelevance.score take input, output and an optional context, while ContextPrecision.score additionally requires expected_output and a mandatory context.

A small trap to close this section: the llm_judges directory contains a factuality subpackage, but the import of the Factuality class is commented out in the code and the name does not appear in __all__. You cannot import a metric by that name from opik.evaluation.metrics, even though the files ship in the package.

Self-hosting versus the cloud, and what stays behind a paid plan

The self-hosted variant stands up a full set of services. The docker-compose.yaml file in the repository shows MySQL 8.4.2, Redis 7.2.4, ClickHouse in release 26.3.16.16 together with ZooKeeper 3.9.4, MinIO as object storage, a Java backend with Liquibase migrations, the dashboard and an OpenTelemetry collector. That is six to eight containers and two databases with completely different operational profiles. There is also a Helm chart in deployment/helm_chart/opik. Let us name it plainly: self-hosting an evaluation platform is one more production service to maintain, ClickHouse backups included.

I read the split of features between variants from the comparison table on the Comet pricing page, checked on 22 August 2026.

Two conclusions come out of that table and they point in opposite directions. The first matches expectation: role-based access control, single sign-on over OAuth 2.0, SAML and LDAP, service accounts, view-only users and compliance with SOC 2, ISO 27001, ISO 9001, HIPAA and GDPR are reserved for the Enterprise plan and are absent both from the self-hosted variant and from the paid Pro plan. The second runs against expectation: guardrails, meaning blocking content on the model's input and output, is marked available only in the open source column, and all three cloud plans carry a dash in that row. In the Python package this corresponds to the opik.guardrails module with the classes Guardrail, Topic, PII, LLMJudge, PromptInjection and CustomGuardrail. What the self-hosted variant lacks is the OpikAssist assistant and the Opik Connect harness, that is, the features where Comet supplies model calls of its own.

The billing unit is the span, defined on the page as a single input and output pair: a model call, a tool call or a tracked function. Not tokens. One span can contain many of them.

Two things need a caveat. First, the same page gives two different numbers about team size: the plan cards speak of up to 10 people on the Free plan and up to 50 on Pro, while the questions and answers section below claims all plans include unlimited team members. I report both and flag the discrepancy, because the page alone cannot settle it. Second, the pricing page does not say what happens after you exceed 25 thousand spans on the free plan. All that is known is that extra spans cannot be purchased on that plan, since the top-up row carries a dash. Whether traffic is dropped or the workspace is blocked, the page does not specify and I will not guess. No annual price is published at all, only the monthly figure, so there is no arithmetic to check here.

Opik against Langfuse, Braintrust, Promptfoo and Ragas

This collection already holds several neighbours, and the differences between them are concrete rather than cosmetic.

Opik sits precisely in the middle of that field. It has open code and a self-hosted variant like Langfuse, but it puts the same weight on evaluation as Braintrust does, with test suites, assertions and experiment comparisons as first-class features rather than an add-on to tracing. It is heavier than Promptfoo, which needs neither an account nor a server, and broader than Ragas, which is a metrics library for retrieval-based systems rather than a platform. Opik in fact wraps Ragas, since its metrics include a RagasMetricWrapper class.

If you want the answer in one sentence: take Promptfoo for quick prompt regression inside a repository, Ragas for computing retrieval metrics in a notebook, Langfuse for traces alone, Braintrust when a closed platform does not bother you, and Opik when you want traces and evaluation in one place and care about the option of running it yourself. It is also worth looking at LangSmith, which occupies a similar niche on the closed side.

The judge model is orthogonal to the platform choice. By default it goes to OpenAI, but through LiteLLM you can plug in Claude or any other provider, and that decision, not the choice of dashboard, will drive the evaluation bill.

Common mistakes

Installing the integration packages alongside the core. npm install opik@2.2.36 opik-openai@2.2.36 ends in a conflict, because the integration declares a peer dependency on opik in the range ^1.8.61. The workarounds are a force flag or staying on the older core line, but first check whether the range was fixed in a newer release.

Assuming the self-hosted variant has everything. Role-based access control and single sign-on are not there, and they are not in the paid Pro plan either. If you need both, the only path is a sales conversation.

Running a judge metric without naming a model. Hallucination() with no arguments reaches for openai/gpt-5-nano from the configuration. On a five thousand item dataset with three judge metrics that is fifteen thousand model calls nobody consciously agreed to.

Forgetting about track=True on metrics. Every metric result is logged as a separate trace by default, so evaluation inflates the span counter your cloud bill depends on. On large runs, track=False is often the sensible setting.

Treating dataset.insert as idempotent. Every call creates a new dataset version, and with parallel batch uploads there is no rollback, so a partial failure leaves the dataset in an intermediate state.

Relying on project_name in evaluate. The argument is deprecated and is ignored when the dataset carries its own project_name. Your traces then land somewhere other than you expect.

Counting on text metrics working right after pip install opik. SentenceBLEU needs nltk, Readability needs textstat, and BERTScore needs bert_score plus a model weights download. None of those is a dependency of Opik.

FAQ

Is Opik fully open source?

Based on the files I checked, yes. The repository has one LICENSE file under Apache 2.0, identical copies sit in both client library directories, and the server directory paths I checked carry no separate license files. I note the caveat that I checked selected paths rather than the whole repository tree.

What does the hosted version cost and what does the free plan give?

The free cloud plan gives 25 thousand spans per month and 60 days of retention, with no option to buy more spans. The Pro plan costs 19 dollars per month, gives 100 thousand spans, top-ups cost 5 dollars per additional 100 thousand, and extending retention to 400 days costs 29 dollars per 100 thousand spans. What exactly happens after the free plan limit is exceeded, the pricing page does not specify.

Which metrics generate a bill at the model provider?

Everything under the llm_judges directory, meaning among others Hallucination, AnswerRelevance, ContextPrecision, Moderation, TrajectoryAccuracy and the whole GEval family. Metrics from the heuristics directory, for example Equals, RegexMatch, IsJson, ROUGE and LevenshteinRatio, compute locally. BERTScore stands apart: it calls no paid interface but downloads and runs a transformer model.

Can Opik be used without sending data outside?

You can stand up the dashboard and databases yourself with ./opik.sh and point the client at a local address through opik.configure(use_local=True). Remember, though, that judge metrics still send content to a model provider unless you plug in a local model, and that the Python package lists sentry_sdk among its mandatory dependencies.

Is Opik worth choosing over Langfuse?

It depends on whether evaluation is your primary concern. If you mainly need traces, both platforms provide them. If you need test datasets, a library of judge metrics and experiment comparisons in the same dashboard, Opik has that ready. On the other hand it is a young product developed by a single company, so vendor lock-in remains a real risk even under an Apache 2.0 license.