DSPy, the end of hand tuning prompts

DSPy, the end of hand tuning prompts

Typical prompt work goes like this: you write an instruction, test it on a few examples, add a sentence, test again. Two days later you have a page and a half of text nobody wants to touch, because nobody knows which sentence does what.

DSPy proposes something else. You describe what goes in and what comes out, pick a strategy, define a quality measure, and supply examples. An optimiser then chooses the instruction and the examples in the prompt, guided by that measure. The library came out of a Stanford University research group and is open source under the MIT licence.

A signature instead of a prompt

A signature describes a contract: which fields go in, which come out, and what they mean. It contains no instruction, since that is meant to emerge later.

pip install dspy

import dspy

class Classification(dspy.Signature):
    """Classifies a customer ticket into a category and priority."""

    content: str = dspy.InputField(desc="the customer's ticket text")
    category: str = dspy.OutputField(desc="one of: login, payments, delivery, other")
    priority: int = dspy.OutputField(desc="from 1 to 5, where 5 is most urgent")

classifier = dspy.Predict(Classification)
result = classifier(content="I cannot log in since yesterday, urgent")
print(result.category, result.priority)

Underneath, the library builds a prompt from the field descriptions, sends it to the model, and parses the response into the declared fields. You write not one sentence of instruction yourself.

That carries two advantages visible immediately. The first is legibility: a signature says what this part does in five lines rather than in a paragraph of prose. The second is the ability to change strategy without touching the rest of the code.

Modules, meaning strategies

A module states how the model should reach an answer. The same signature with a different module behaves differently.

simple = dspy.Predict(Classification)
with_reasoning = dspy.ChainOfThought(Classification)
with_tools = dspy.ReAct(Classification, tools=[check_history])

The first asks directly. The second has the model write out its reasoning first, which improves accuracy on multi step tasks at a cost in tokens. The third lets the model call tools and work iteratively.

Changing strategy is swapping one line rather than rewriting a prompt. That is what separates this approach from working with text: with a hand written prompt, adding reasoning means reworking the instruction and risking breaking something else along the way.

Modules compose into programs, where one's output feeds another. A program is an ordinary class, so all of Python applies inside it: conditions, loops, error handling.

The metric, the thing that matters most

The whole library rests on one assumption: you can measure whether an answer is good. Without a metric the optimiser has nothing to optimise.

def correctness(example, prediction, trace=None) -> float:
    category_hit = example.category == prediction.category
    priority_close = abs(example.priority - prediction.priority) <= 1
    return (category_hit + priority_close) / 2

A metric can be a simple comparison, a sum of several conditions, or a score from another model when checking things no direct comparison covers, such as factual grounding.

Here lies the real difficulty of this approach and it deserves naming plainly. If a task has an unambiguously correct answer, the metric is obvious and the library delivers a lot. If the task is writing pleasant prose, there is no metric, and without one optimisation is turning knobs blind.

The example set must also be built. Fifty cases with expected results is a sensible minimum, two hundred for harder tasks. That is work, only moved from editing a prompt to preparing data.

Optimisers

With a module, a metric, and examples, you run an optimiser. The result is the same program with chosen instructions and examples in its prompt.

from dspy.teleprompt import MIPROv2

optimiser = MIPROv2(metric=correctness, auto="medium")
optimised = optimiser.compile(
    classifier,
    trainset=training_examples,
    valset=validation_examples,
)

optimised.save("classifier.json")

Several approaches are available. The simplest picks prompt examples from among the supplied cases. More elaborate ones search the space of instructions and examples together, scoring combinations against a validation set.

A newer approach is a reflective optimiser that analyses full execution traces, identifies which step failed, and proposes an improved instruction for that specific step. On multi stage programs that delivers more than optimising everything at once, since the problem usually sits in one place.

You save the result to a file and load it in production, so optimisation is a build step rather than something happening on every request. That file belongs in the repository next to the code, because then you know which version of the program matches which version of the application.

Multi stage programs

A single signature suffices for a task describable with one input and one output. Real applications rarely are, so modules compose into a program.

class AnswerQuestion(dspy.Module):
    def __init__(self):
        self.rephrase = dspy.ChainOfThought("question -> search_query")
        self.answer = dspy.ChainOfThought("context, question -> answer")

    def forward(self, question: str):
        query = self.rephrase(question=question).search_query
        fragments = self.store.search(query, limit=5)
        return self.answer(context=fragments, question=question)

The shorthand in quotes is a signature written on one line, handy for simple steps. Searching a vector database such as Chroma is an ordinary call here, since a program is normal code.

The biggest advantage appears at optimisation. The optimiser tunes both steps together, guided by the quality of the final answer rather than of the intermediate query. That matters, because the best search query does not always look the way a person would design it.

With that arrangement, think about a metric for the whole from the start. Scoring an intermediate step is tempting because it is easier, only optimising against it improves something that is not the goal.

Evaluation and comparing versions

Beyond optimisation the library offers a measurement tool worth using even without an optimiser.

from dspy.evaluate import Evaluate

evaluation = Evaluate(devset=validation_examples, metric=correctness, num_threads=8)
score = evaluation(optimised)
print(f"Accuracy: {score}")

That one number compares versions against each other: before and after optimisation, on a cheaper and a stronger model, with reasoning and without. Without it, choosing a variant rests on impressions from a few manual checks.

Run the evaluation in the build pipeline too, exactly like ordinary tests. A vendor's model change or a library update can lower accuracy, and without measurement you notice only after user complaints.

A practical note on cost: evaluating two hundred examples is two hundred calls, so on every commit that is too much. A sensible split runs a small set on every change and the full one before a release.

What it costs

Optimisation means many model calls: every combination of instruction and examples is scored against the validation set. With two hundred examples and dozens of combinations that runs to thousands of calls.

Three approaches limit that cost. The first is optimising on a cheaper model and carrying the result to a stronger one, which often works, since a chosen instruction transfers. The second is a smaller validation set, since fifty examples usually suffice to compare combinations. The third is an economical mode capping the number of attempts.

Price that cost before running, multiplying call count by the rate. Sometimes a one off optimisation costs as much as a month of running the system, and then you must decide whether the improvement justifies it.

Set that estimate against the real bill afterwards, since an optimiser usually makes more calls than simple multiplication suggests. The cheapest way to see it is a proxy layer recording every call together with its price, with no code change beyond the base URL. Helicone worked exactly that way, but after the Mintlify acquisition in March 2026 it went into maintenance mode, so for a new project you have to check what in this category is still being developed.

When it pays off and when it does not

It pays off on measurable, repeatable tasks: classification, extracting data into a structure, answering questions from supplied context, transformations with a checkable result. There an optimiser usually finds an arrangement better than a person would write.

It also pays off when changing models. A prompt polished for one model rarely works as well on another, and re optimising is one command rather than two days of editing.

It does not pay off on one off tasks, on text judged subjectively, and on simple applications where an ordinary prompt is good enough. The entry cost covers preparing a set and writing a metric, so with one prompt in an application that effort will not repay itself.

The third case for holding off is unwillingness to accept that the prompt will not be legible to a person. Optimisation output is sometimes odd: an instruction with unobvious phrasing and examples chosen in a way nobody would have thought of. It works better and it is harder to discuss.

Building the example set

This is the most laborious part and the one deciding the result. A few rules shorten it considerably.

The first: take examples from real traffic rather than inventing them. Cases invented at a desk are too clean and skip exactly what causes trouble in practice: typos, sentences cut off midway, two questions in one message.

The second: include the cases where the current solution fails. Those carry the most information, since a set made only of easy cases shows no difference between variants.

The third: keep proportions close to reality. If ninety percent of tickets in practice concern one category, a set split evenly across five categories optimises for a distribution that does not occur.

The fourth: separate the training set from the validation set before the first run. A split of seventy to thirty suffices, and mixing them yields a result that looks good and says nothing.

examples = [
    dspy.Example(content=c, category=k, priority=p).with_inputs("content")
    for c, k, p in production_data
]

training, validation = examples[:140], examples[140:]

The method naming input fields is mandatory here, since without it the library cannot tell what is input and what is expected output.

DSPy against the alternatives

These approaches do not exclude each other. You can build an agent in an agent library and optimise a single accuracy critical step, classification for instance, here, then load the finished result.

Note too that the benefit is largest where a task repeats thousands of times a day. A few percentage points of accuracy at that volume means more than at a hundred calls a week.

There is one more gain, rarely discussed and sometimes decisive for longer maintenance. An optimised program stores its instruction in a file rather than in code, so returning to last month's version is a matter of swapping a file. With a prompt typed into code, the same operation means digging through change history and guessing which version was in force while the system worked well.

The situation looks similar when changing model vendor. Instead of rewriting the instruction for the new model, you run the optimisation again and compare the result numerically. That turns a migration decision from a discussion about impressions into a comparison of two numbers.

Common mistakes

The first is a metric measuring something other than what you want. The optimiser fits the program exactly to it, so a measure rewarding long answers yields long answers whether or not they are better.

The second is optimising and validating on the same set. The result looks excellent and says nothing about behaviour on new data, since the program learned those very examples.

The third is too small a set. Twenty examples let an optimiser fit to randomness rather than to regularity.

The fourth is running optimisation without pricing it. Thousands of calls on a strong model can cost more than anybody assumed.

The fifth is treating an optimised program as finished for good. A model change, a shift in data distribution, or a new ticket category all call for another run.

The sixth is reaching for this approach on a task you cannot measure. Without a metric the whole machinery has nothing to optimise, and a hand written prompt is then simpler and more honest.

FAQ

How does DSPy differ from writing prompts?

Rather than writing an instruction, you describe input, output, and a quality measure, and an optimiser chooses the instruction and examples from your data. The entry cost is higher, since a set and a metric must be prepared, and the gain appears on measurable, high volume tasks.

Is DSPy free?

Yes, the library is open source under the MIT licence. You pay only for model calls, though optimisation itself generates many, so it is often the main cost rather than the system's later operation.

Does it work with any model?

Yes, the model layer is vendor neutral and covers Claude, OpenAI models, and local models served through Ollama, among others. Changing models usually calls for re optimisation, since a chosen instruction is fitted to a specific model.

How many examples are needed?

Fifty is a sensible minimum on a classification task, two hundred on a harder one. Quality matters more than count: examples should come from real traffic and cover the cases where the system has been failing.

DSPy or an ordinary agent library?

The choice is not exclusive. An agent library organises flow and integrations, while this approach optimises a single accuracy critical step. A common arrangement is an agent built elsewhere with one module optimised here inside it.

Documentation sits on the project site, and the source code in the GitHub repository.