Haystack, an LLM application assembled from blocks

Haystack, an LLM application assembled from blocks

An application answering questions from your own documents involves several steps: turn the question into a vector, find fragments, order them, build a prompt, call the model. Written directly it is a hundred line function where changing one stage means reading the whole thing.

Haystack breaks that into components wired into a graph. Every step is its own class with declared inputs and outputs, and the connections between them are stated explicitly. The effect is that swapping the embedding model or adding a filter touches one place rather than the entire function.

Component and pipeline

A component is a decorated class declaring output types with a method doing the work.

pip install haystack-ai sentence-transformers-haystack

from haystack import Pipeline
from haystack_integrations.components.embedders.sentence_transformers import SentenceTransformersTextEmbedder
from haystack.components.builders import ChatPromptBuilder
from haystack.components.generators.chat import OpenAIChatGenerator

pipeline = Pipeline()
pipeline.add_component("embedder", SentenceTransformersTextEmbedder())
pipeline.add_component("retriever", retriever)
pipeline.add_component("prompt", ChatPromptBuilder(template=template))
pipeline.add_component("model", OpenAIChatGenerator(model="gpt-5-mini"))

pipeline.connect("embedder.embedding", "retriever.query_embedding")
pipeline.connect("retriever.documents", "prompt.documents")
pipeline.connect("prompt.prompt", "model.messages")

result = pipeline.run({"embedder": {"text": "how do I download an invoice copy"}})

One thing in that example changed with the third release, from July 2026. The components built on sentence-transformers moved out of the core into a separate package, along with thirty others, so they now need a separate install and a different import path. Code written for the second version fails right there with a missing module error.

Connections are explicit and that is the heart of this approach. You see what feeds what, and wiring an output to an input of an incompatible type fails when the pipeline is built rather than on first run.

That separates the framework from libraries where data flow is implicit and follows from call order. There, legibility depends on the author's discipline; here it follows from structure.

The price of that explicitness shows immediately: more lines for the simplest case. On a three step pipeline a plain function would be shorter, and the advantage appears only at ten.

Branching and loops

The framework's second version added support for graphs with branches and cycles, which widens what can be expressed.

A branch routes a query down a different path depending on its kind. A documentation question goes to search, an order status question to a programming interface, and an out of scope question to a refusal before anything costly runs.

A cycle repeats a step until a condition holds. The typical use refines an answer that failed a check: the model generates, an evaluating component checks, and on failure the prompt returns to the model with a note about what was wrong.

Mind one thing here. A cycle without an iteration cap can spin indefinitely, burning tokens without finishing. The cap is set at definition time and is a matter of common sense rather than an option.

Configuration in a file

A pipeline serialises to a file and loads from one, which carries consequences beyond convenience.

components:
  embedder:
    type: haystack_integrations.components.embedders.sentence_transformers.sentence_transformers_text_embedder.SentenceTransformersTextEmbedder
    init_parameters:
      model: intfloat/multilingual-e5-base
  model:
    type: haystack.components.generators.chat.OpenAIChatGenerator
    init_parameters:
      model: gpt-5-mini

connections:
  - sender: embedder.embedding
    receiver: retriever.query_embedding

That lets you change a model or parameters without deploying a new application version, which helps during tuning. It also lets you keep different pipeline variants per environment in files rather than in conditions scattered through the code.

The trap is the same as with any configuration outside code. System behaviour then depends on a file whose changes go through no review and carry no tests. A sensible compromise keeps those files in the repository and treats changes to them exactly like code changes.

Agents

Newer versions add an agent component that fits inside a pipeline alongside the rest. That means one flow can contain a deterministic step and a step where the model decides the order of actions itself.

That arrangement is practical, since it matches how most applications look. Retrieving fragments, filtering by permissions, and formatting an answer must happen identically every time. Deciding whether to reach for an extra tool, and which, needs judgement about content.

Handing everything to the model would waste money, since some steps need no judgement and every call costs. Writing everything as a deterministic pipeline will not work, since some decisions depend on the query's content.

The practical rule: steps that must run identically stay components, and an agent goes where judgement is needed. That is the same principle as with LangGraph and other graph based solutions.

Indexing documents

The answering pipeline is half the system. The other half prepares documents, and it decides quality more than anything on the answering side.

indexing = Pipeline()
indexing.add_component("converter", PyPDFToDocument())
indexing.add_component("cleaner", DocumentCleaner())
indexing.add_component("splitter", DocumentSplitter(split_by="sentence", split_length=5))
indexing.add_component("embedder", SentenceTransformersDocumentEmbedder())
indexing.add_component("writer", DocumentWriter(document_store=store))

indexing.connect("converter", "cleaner")
indexing.connect("cleaner", "splitter")
indexing.connect("splitter", "embedder")
indexing.connect("embedder", "writer")

The cleaning step strips repeated headers, footers, and empty lines. That sounds minor, and on documents carrying page numbers in every fragment it can lower accuracy noticeably, since the embedding model treats that noise as part of the content.

The splitting step matters most and deserves experimentation. Sentence splitting with overlap suits continuous prose, while technical documentation does better split by headings, since a section is the natural unit of an answer.

Record the section title and source URL in metadata. Without them an answer cannot point at its origin and a user cannot verify it.

The last thing is identifiers. Built from the source URL and fragment number they let a document be reindexed without creating duplicates, which for documentation changing weekly is a condition rather than a convenience.

Quality evaluation

The framework ships evaluation components that plug into a pipeline like any other.

The measures are those covered in the piece on Ragas: whether an answer rests on the context, whether retrieved fragments are relevant, whether the answer addresses the question. Wiring them into the pipeline lets you measure those things across a case set after every change.

The value of that arrangement is that evaluation uses the same pipeline as production. With a separate tool there is always a risk of measuring something other than what actually runs, since the configuration drifted.

The opposite approach scores from outside, without wiring components into the pipeline. TruLens instruments the application and computes the same triad of measures under an MIT licence, which can be decisive, since some tools in this category carry licences restricting commercial redistribution. The library came out of Truera, acquired by Snowflake, so plenty of documentation examples assume that vendor's services, though it runs locally and requires no account.

Mind the cost, though. Judge model evaluation means extra calls, so a full set on every code change is financially unrealistic. A sensible split runs a small set often and the full one before a release.

Custom components

The built in set covers typical steps, and things specific to your domain you write yourself. That is simpler than it looks.

from haystack import component
from typing import List

@component
class PermissionFilter:
    @component.output_types(documents=List[Document])
    def run(self, documents: List[Document], user_id: str):
        allowed = [d for d in documents if d.meta["owner"] == user_id]
        return {"documents": allowed}

A decorator and an output type declaration are all a component needs to plug into a pipeline like any other. Build time type checking covers it exactly as it covers built in ones.

That particular example deserves consideration whatever the framework. Filtering by owner after retrieval rather than inside the database query is slower and less safe, since the database then returns other people's fragments and the sieving happens later. The filter belongs in the query, and a component like the above is a safeguard in case somebody omits it.

The second typical use covers transformations specific to your document format: extracting fields from a structure, normalising names, attaching context from an external system. Such steps rarely have ready versions, since they depend on how your data looks.

The third is integrating a service with no component available. Wrapping an interface call in a component takes a dozen or so lines, and the gain is that you work with it exactly as with the rest of the pipeline.

Haystack against the alternatives

The last row is unfairly overlooked. A flow of an embedding call, a database query, and a model call is thirty lines anybody can read. A framework starts paying off with branching, component swapping, and a need for evaluation.

Choosing among the first three rows depends on what you value. Explicit connections and build time type checking are the first row's advantage, mattering on a pipeline somebody will read in six months. Ready integration count is the second's, mattering when connecting many sources.

It is worth naming the thing that gets lost in such comparisons, though. The quality of a system answering from documents is decided by how documents are split, which embedding model is chosen, and whether filters exist, not by the framework. Those three carry across libraries unchanged, so the work put into them does not perish in a migration, while the work put into learning a framework does.

The practical conclusion is to start with the simplest option and measure relevance on your own questions. Only once you know where the problem sits should you reach for a tool giving more control, since otherwise you add complexity without knowing whether it fixes anything.

Production deployment

A pipeline run from a script suits trials. A deployment serving traffic needs several decisions.

The first concerns startup. A pipeline created per request loads the embedding model every time, which with a local model means several seconds. A pipeline built once at application start and kept around removes that cost.

The second is concurrency. Components using a local model do not always handle many requests at once, so at higher traffic retrieval and generation deserve splitting into separate services, with the model hosted on a server built for concurrency.

The third is error handling. An unavailable vector database or a rate limit at a model vendor breaks the pipeline, so decide what the user sees: a message about a temporary problem, or an answer without context clearly marked incomplete.

The fourth is traces. A pipeline of six components is transparent at build time and opaque during diagnosis unless you record what passed through which step. Wiring in an observability tool such as Langfuse turns "why is this answer bad" from a guess into a check.

Common mistakes

The first is building a pipeline for a task that is one call. Three components for something fitting in ten lines add complexity with no benefit.

The second is a cycle without an iteration cap. A loop refining an answer until it satisfies may never satisfy, and token usage grows linearly with attempts.

The third is configuration in a file outside the repository. System behaviour then depends on something nobody reviews and that carries no change history.

The fourth is skipping evaluation. Changing the embedding model or the prompt template usually improves some cases and breaks others, and without a test set only the first half is visible.

The fifth is measuring answer quality alone. Without retrieval metrics you cannot tell whether to fix the prompt or the document splitting, and those are entirely different jobs.

The sixth is treating built in components as the only option. A custom component is a decorated class with two methods, so adding a step specific to your domain is a quarter of an hour's work.

FAQ

How does Haystack differ from LangChain?

It emphasises explicitness: connections between components are declared directly, and type mismatches surface when the pipeline is built. LangChain carries far more ready integrations and starts faster, at the cost of more abstraction to learn.

Is Haystack only for search?

No, though that is its strongest side. Newer versions carry an agent component fitting inside a pipeline alongside the rest, so one flow can combine deterministic steps with steps where the model decides for itself.

Must configuration live in a file?

No, a pipeline can be built entirely in code. A file helps with tuning without deployment and with per environment variants, and it belongs in the repository so changes go through review.

How does it work with vector databases?

Through document store components available for the popular options, Qdrant and Chroma included. Switching databases then means swapping a component without touching the rest of the pipeline.

When is it not worth reaching for?

On a flow of two or three steps without branching. Thirty lines of code is clearer then than a pipeline of components, and the framework starts paying off with branching and with a need to swap parts without disturbing the rest.

Documentation sits on the project site, and the source code in the GitHub repository.