Promptfoo, tests for things ordinary testing cannot cover

Promptfoo, tests for things ordinary testing cannot cover

Code is tested with assertions because its output is deterministic. A model's answer is not, so the question "did that prompt change improve anything" usually ends with checking three examples and a feeling that it is better.

Promptfoo turns that feeling into a number. You define test cases, state what must hold, and run a comparison across many variants at once: different prompts, different models, different settings. The tool's second half generates adversarial probes checking whether the application can be broken.

In March 2026 OpenAI announced its acquisition of the project, with assurances that it stays open source under the current licence and that the technology moves into that vendor's platform. For users of the open source version nothing changed.

Your first test suite

Configuration lives in a file, so it goes under version control like the rest of the code.

npx promptfoo@latest init

prompts:
  - "Classify this ticket into a category: {{content}}"
  - file://prompts/classification-v2.txt

providers:
  - openai:gpt-5-mini
  - anthropic:claude-sonnet-5

tests:
  - vars:
      content: "I cannot log in since yesterday"
    assert:
      - type: contains
        value: "login"
      - type: latency
        threshold: 3000

  - vars:
      content: "Please resend my invoice for March"
    assert:
      - type: contains
        value: "payments"

A run compares every prompt against every model on every case, so two prompts and two models across two cases produce eight calls. Results appear in a table showing at a glance which variant passes more tests.

That construction is the heart of the tool. Rather than fixing one prompt repeatedly, you compare several variants at once on the same set and pick the one performing best by a measure rather than by impression.

Kinds of assertion

Assertions fall into three groups differing in cost and reliability, and mixing them without discrimination is the commonest mistake.

Deterministic assertions check facts about the response: whether it contains a string, matches a pattern, is valid JSON, stays within a time or cost limit. They are free, instant, and should make up most of a suite.

assert:
  - type: is-json
    value:
      required: [category, priority]
  - type: javascript
    value: "output.priority >= 1 && output.priority <= 5"
  - type: cost
    threshold: 0.01

Model based assertions check things no direct comparison covers: whether an answer stays with the supplied facts, carries the right tone, promises nothing forbidden. They cost tokens and add noise of their own, so reserve them for what no simpler check can cover.

assert:
  - type: llm-rubric
    value: "The answer rests solely on the supplied context and adds no facts"
  - type: factuality
    value: "Invoice copies are downloaded from the billing panel"

The third group compares against a reference answer, useful when you have references prepared. They are more reliable than free scoring and require work to build the set.

Red teaming

The tool's second half checks whether an application can be pushed into behaviour you do not want. That is hard to do by hand, since it demands inventiveness and systematicity at once.

npx promptfoo@latest redteam init
npx promptfoo@latest redteam run

The tool generates attack attempts across several categories. Prompt injection checks whether user supplied content can override the system instruction. Jailbreaking hunts for phrasings that make the model do what it was told to refuse. Data leakage checks whether the system instruction or another user's data can be extracted. Tool misuse checks whether an agent can be led into calling an operation outside its scope.

The output is a report with the specific queries that worked. That matters, since a vulnerability list without an example is useless, while a query that actually got through can go straight into the test suite as a regression case.

Run this before a release rather than after an incident. An application with an agent holding write access is a special case here, since a successful attack's consequences reach beyond an inappropriate answer.

Testing the whole path, not the prompt alone

The commonest methodological mistake is testing an instruction in isolation from the rest of the application. A prompt with hardcoded context passes its tests while the application answers badly, because the problem sits in which fragments reach the context.

The tool can call your own endpoint, so you test what actually runs for the user.

providers:
  - id: https://localhost:3000/api/assistant
    config:
      method: POST
      headers:
        Content-Type: application/json
      body:
        question: "{{content}}"
        userId: "test-118"
      transformResponse: json.answer

With that arrangement a test covers vector search, tool calls, and all the logic in between. If the application pulls fragments from Chroma or from the pgvector extension, that stage is usually responsible for most bad answers rather than the instruction's wording.

Test the retrieval stage separately too. A set of questions with the fragment that should reach the context marked lets you measure retrieval accuracy independently of answer quality. That separates two problems which otherwise blur together.

Team work and reviewing results

Test results in a terminal suit one person and are useless for a team. The tool offers a browser view with a comparison table showing answers side by side.

npx promptfoo@latest view

That view helps most with manual scoring. Some things are hard to express as an assertion and easy to judge by eye, so reviewing twenty answers in a table takes a quarter of an hour and says more than a pass count.

In team work, settled roles matter too. Whoever changes the prompt runs the suite, but judging whether the change is acceptable should belong to somebody who knows the business requirements. That is the same principle as in code review.

Settle who adds new cases as well. The best source is user complaints and situations where the application failed, so whoever handles complaints is the natural candidate, provided the suite format is simple enough to use without knowing the code.

Use in a build pipeline

A suite run by hand works for two weeks and then stops. Wiring it into the build makes a regression surface on the change rather than at the user.

- name: Prompt tests
  env:
    PROMPTFOO_PASS_RATE_THRESHOLD: '90'
  run: npx promptfoo@latest eval --config promptfoo.yaml

Two things need settling in such a rollout. The first is cost: a full suite on every commit is hundreds of calls a day, so a sensible split runs a small fast set on every change and the full one before a release or once a day.

The second is nondeterminism. A test on a model can pass four times and fail the fifth, so set the pass threshold at suite level rather than demanding a clean sweep. The environment variable shown above does that, taking a percentage of passing cases; it defaults to a hundred, and the command exits with code one hundred once the result drops below the threshold.

Keep results between runs, since only comparison with the previous one shows direction. A single result says how many tests passed; the difference says whether the last change helped.

Pricing

The price list publishes no figure other than zero: both paid plans are quoted individually against team size. The free version's single limit is worth knowing, though. A probe is one request sent to the system under test during red teaming, and the Community plan covers ten thousand probes a month at no charge. Additional probes are bought only under an Enterprise agreement, so at larger testing volumes it is that cap, rather than the tool's price, that decides when the conversation with a salesperson starts.

The library itself is free, but testing costs tokens. A hundred cases against two models and two prompts is four hundred calls, so price a large suite before running it.

Three approaches limit that cost. The first is favouring deterministic assertions, which cost nothing. The second is testing on a cheaper model where you check prompt logic rather than answer quality. The third is caching results, so a rerun with no changes generates no calls.

What to do with red teaming output

A report listing successful attacks is where the work starts rather than ends. Three things need settling for every vulnerability found.

The first is consequence. An answer where the model strayed off topic is a reputational nuisance. An answer where an agent called an operation changing somebody else's account is a security incident. The same attack technique can produce either, depending on what the agent has to hand.

The second is where the fix belongs. Strengthening the system instruction helps partly and can be worked around, since a model can always be talked round. The durable answer sits in code: permission checks inside the tool, narrowed operation scope, required approval on irreversible actions.

There is sometimes a third place: a filter inspecting the model's input and output separately from its instruction, running on every request. The best known open library of that kind is LLM Guard from Protect AI, and it needs a caveat, because the project was archived in July 2026 and its maintainers wrote plainly that no new detectors are coming. The code still installs and still runs, but putting a library that will get no fixes into your stack is a decision to take with open eyes. Treat the filter itself as a layer that reduces how many attempts succeed, not as a substitute for permission checks inside the tool.

The third is recording it in the suite. A query that got through goes in as a regression case, so the same hole does not return on the next prompt change. Without that step a fix lasts until the next substantial edit.

Remember too that automated red teaming checks patterns known to the tool. It does not replace reviewing what the agent can do at all, and that question deserves asking before anybody starts testing. An agent with no access to data changing operations is immune to a whole category of problems by definition.

Promptfoo against the alternatives

These tools serve different stages and two are usually used together. This one compares variants before release; an observability tool diagnoses what happens in production. Production traces are in fact the best source of test cases, so the two form a loop.

The last row is a different category: rather than comparing human written variants, it chooses the instruction automatically. They combine, using this tool as an independent check on the optimisation result.

Note that none of these tools replaces thinking about what should actually be measured. A test set reflects what you decided mattered, so if it lacks the cases where the application genuinely fails, a high score means nothing. It is the same problem as test coverage in ordinary code: the number goes up and the bugs stay.

Common mistakes

The first is a suite made of easy cases. Tests that always pass show no difference between variants, which is exactly why the suite exists.

The second is model based assertions where a value comparison suffices. Checking whether a response is valid JSON with the required fields costs nothing, while scoring the same thing with a model costs tokens and is sometimes unstable.

The third is testing the prompt alone rather than the whole path. If the application pulls fragments from a vector database, a test on hardcoded context misses the retrieval problem responsible in practice for most bad answers.

The fourth is skipping red teaming on an application accepting user content. Prompt injection is not a theoretical threat, particularly when an agent holds tools that change data.

The fifth is running tests only by hand. A suite outside the build stops being used within a month, and then the user finds the regression.

The sixth is demanding a hundred percent pass rate. With nondeterministic answers that threshold makes a team start disabling tests rather than improving the prompt.

FAQ

Is Promptfoo free?

Yes, the open source version under the MIT licence covers full testing and red teaming and you run it yourself. The Community plan is free forever with one limit: ten thousand red teaming probes a month. The Enterprise and Enterprise On-Premise plans are quoted individually, with no published rates. You do pay for the tokens the tests consume.

What does the OpenAI acquisition mean?

In March 2026 OpenAI announced the acquisition, stating the project stays open source under its current licence and that existing customers continue to be supported. The technology is to move into that vendor's agent platform. For open source users it changes nothing about how they work, though a change of owner is worth weighing when planning years ahead.

How does it differ from observability tools?

This tool checks variants before deployment against a prepared case set. Tools like Langfuse collect traces from a running application and show what actually happens. They work best together: traces supply cases for the suite, and the suite keeps a fix from breaking something else.

Can it test agents rather than only prompts?

Yes, the tool also calls your own endpoint, so you test the whole path including retrieval and tool calls. That is usually more sensible than testing the prompt alone, since problems most often sit in the context rather than in the instruction.

Where should I start?

With twenty cases taken from real traffic, including situations where the application failed. Add deterministic assertions, run a comparison of two prompt variants, and only then expand the suite with model scoring where a simpler check falls short.

Documentation sits on the project site, and the acquisition was announced on the Promptfoo blog.