Chroma, a vector database you can start in a minute

Chroma, a vector database you can start in a minute

Most vector databases begin with creating an account, picking a region, and configuring an index. Chroma begins with installing a package and one line of code, and the database lives in a file next to your project.

That approach made it the default choice for learning and prototyping for years. The change of recent years is that release 1.0 in April 2025 rewrote the core in Rust, giving a fourfold performance gain, and that the hosted version launched in August 2025. A toy has become something you can put production on.

What a vector database is for

Keyword search finds documents containing a given word. Vector search finds documents of similar meaning, even when they share no word.

The mechanism is simple to describe. Text becomes a vector of numbers representing its meaning, and a question becomes one the same way. The database returns fragments whose vectors sit closest to the question's vector. A sentence about changing a password and one about recovering account access land near each other despite sharing no word.

The most common use is supplying a model with context from your own documents. The model does not know your internal documentation, so before answering you pull the three best matching fragments from the database and append them to the prompt. That construction sits behind most assistants answering questions about company content.

Getting started

pip install chromadb

import chromadb

client = chromadb.PersistentClient(path="./store")
collection = client.get_or_create_collection("documentation")

collection.add(
    documents=[
        "Change your password in account settings, Security tab",
        "Download invoices from the billing panel, Payment history section",
        "Delete your account in settings, the operation is irreversible",
    ],
    ids=["doc-1", "doc-2", "doc-3"],
)

result = collection.query(query_texts=["how do I recover access"], n_results=2)
print(result["documents"][0])

There is no embedding step here, since the library handles it itself, by default with a local model downloaded on first use. That is convenient to start with and fine for a prototype, but a serious deployment deserves a deliberate model choice.

The persistent client writes data to the given directory, so it survives a process restart. The variant without a path keeps everything in memory and disappears with the program, which suits tests and nothing else.

Metadata, where quality is won

Vector similarity alone rarely suffices. In practice you also need filters: only this user's documents, only from the last year, only in a chosen language.

collection.add(
    documents=[text],
    metadatas=[{"type": "faq", "language": "en", "year": 2026}],
    ids=["faq-118"],
)

result = collection.query(
    query_texts=["how do I change my plan"],
    n_results=3,
    where={"$and": [{"language": "en"}, {"year": {"$gte": 2025}}]},
)

The filter applies before search, so it narrows the set rather than sieving results afterwards. That difference is fundamental for correctness: without the mechanism another customer's document can rank above the right one, because it matches better semantically.

Plan metadata before you fill the database. Adding a field to a hundred thousand entries means regenerating embeddings or a bulk update, and both cost. Four fields worth handling from the start are owner identifier, document source, date, and language.

Splitting documents into fragments

Do not put whole files into the database. A vector computed for a twenty page document describes everything at once and therefore nothing in particular, and the model then receives a wall of text in which the answer is one sentence.

A sensible starting point is fragments of five hundred to a thousand characters with slight overlap. Overlap guards against cutting the sentence carrying the answer exactly at a fragment boundary.

More important than the number, though, is the splitting rule. Cut along document structure, meaning headings and paragraphs, rather than every fixed number of characters. A fragment matching one documentation section is a better search unit than a slice starting mid sentence.

Each fragment deserves a section title and source URL in its metadata. That way the model's answer can carry a link and the user can check where the information came from.

Choosing an embedding model

The default local model is fast and free, but handles languages other than English less well than multilingual models. That is the most common cause of a database returning off target fragments, with the blame landing on the database rather than on the embeddings.

from chromadb.utils import embedding_functions

embeddings = embedding_functions.SentenceTransformerEmbeddingFunction(
    model_name="intfloat/multilingual-e5-base"
)

collection = client.get_or_create_collection("documentation", embedding_function=embeddings)

The alternative is a cloud vendor's model, billed per token, usually more accurate and more convenient at volume. The choice comes down to three questions: may the data leave your infrastructure, how many fragments will you index, and does quality in a language other than English matter.

One thing here is irreversible. Changing the embedding model requires recomputing the whole collection, since vectors from different models are not comparable. Test two or three models on a sample of a hundred fragments before indexing a hundred thousand.

Full text search and a mixed approach

Meaning similarity alone fails in one recurring case: when the user searches for an exact string. An order number, a function name, or an error code are things a vector cannot capture, since they mean nothing semantically.

Chroma lets you query a collection by document content too, so alongside vector search you get a text filter.

result = collection.query(
    query_texts=["login error"],
    n_results=5,
    where_document={"$contains": "ERR_AUTH_401"},
)

The practical approach combines both paths. You run vector and text search, then merge results, promoting fragments that appeared in both. That simple move lifts accuracy more than swapping the embedding model for a larger one, particularly in technical documentation full of proper names.

It also helps to know when vector search is simply the wrong tool. Asking for the newest entry or for every ticket from a given customer is a query for an ordinary database, not a vector one. Vectors answer questions about similarity, not about facts.

That boundary is a common source of disappointment on a first deployment. A user asks how many orders came in last month, receives a fragment of documentation about orders, and concludes the system does not work. The remedy is recognising the kind of question and routing it to the right source, rather than trying to teach a vector database to count.

Updates and deleting data

A knowledge base is not static. Documents change, and fragments removed from the source must disappear from the index too, otherwise the model answers with content that no longer exists.

collection.upsert(
    ids=["faq-118"],
    documents=[new_text],
    metadatas=[{"type": "faq", "language": "en", "year": 2026}],
)

collection.delete(where={"source": "legacy-documentation"})

The key to order lies in identifiers. If they are generated randomly on every indexing run, reprocessing a document creates duplicates instead of updating entries. Build the identifier from the source URL and fragment number, and reindexing becomes safe.

The second mechanism worth having from the start is deletion by metadata. Since every fragment knows its source, withdrawing a whole document comes down to one call rather than hunting entries by hand.

Chroma Cloud and scale

The local version runs in one process, so a single machine is its limit. It holds up to a few hundred thousand fragments; beyond that memory and index build time start to hurt.

The hosted version uses the same interface but rests on a distributed architecture. Billing follows actual usage, and data sits in tiers of different temperature, where rarely queried fragments drop to cheaper object storage. A database lives in the region you choose and does not leave it, which is sometimes a formal requirement.

Interface compatibility matters most in practice here. A prototype written against the local client moves to the cloud version by changing how the client is created, without touching query logic. Few databases offer that path.

There is a third route, sensible when you do not want to run a database process at all. Upstash exposes vector search over plain HTTP and bills per request, so you can query it from functions running close to the user that do not permit opening a TCP socket at all. The price is fixed: every operation carries HTTP request overhead, so in a loop of several dozen calls the model stops paying off.

Pricing

Both paid rows are a subscription plus usage counted separately: writes, storage, queries, and returned traffic each carry their own rate per gigabyte or terabyte. The credit included with the team plan does not roll over into the next month, so an unused allowance simply expires.

It also pays to know what happens once a limit is crossed, since the mechanism is harsher than it looks. There are two limits, yours and the vendor's, and crossing either one pauses the service until the threshold is raised. The bill does not quietly grow; the database stops answering, which in a production deployment is an event worth an alert.

When estimating cost, remember the item outside the database price list: embeddings. Indexing a million fragments with a paid model can cost more than a month of running the database itself, and it is a one off cost on first indexing and a recurring one on every model change.

How to tell whether search works

This is the most skipped stage in projects built on semantic search. Accuracy usually gets judged on a handful of questions asked by hand, which supports an impression but not a decision.

A simple test set fixes that. Gather thirty real questions and for each record the identifier of the fragment that should appear in the answer. Then count in how many cases that fragment lands in the top three.

hits = sum(
    1 for question, expected in test_set
    if expected in collection.query(query_texts=[question], n_results=3)["ids"][0]
)
print(f"Accuracy: {hits}/{len(test_set)}")

That one number lets you compare two embedding models, two splitting strategies, and the effect of adding filters. Without it every change is a bet, and a regression is noticed first by the user.

Measure two cases separately, since they blur together and have different causes. The right fragment being in the database but ranking low points at embeddings or at the splitting rule. The fragment being absent altogether points at an indexing gap, and no model change will fix that.

Chroma against the alternatives

The last row deserves a comment, since it is often the best choice and the least often considered. If the application uses PostgreSQL anyway and there are tens of thousands of fragments, an extension to the existing database saves an entire separate service to maintain.

The practical order of decisions runs like this. First check whether you need a separate database at all, because with a small collection you do not. Then choose between self hosting and a managed service, guided by who will maintain it. Only at the end compare specific tools, since the differences between them are smaller than the difference between a good and a bad way of splitting documents.

Common mistakes

The first is indexing whole documents rather than fragments. The search result is then technically correct and practically useless, since the model receives twenty pages instead of the right paragraph.

The second is missing filters on data ownership. In a multi user application an unrestricted query returns everyone's fragments, which is a leak rather than an accuracy defect.

The third is changing the embedding model without recomputing the collection. Old and new vectors sit in different spaces, so results turn random, and nothing reports the error.

The fourth is returning too many fragments. Twenty results in the prompt cost tokens and distract the model, while three apt ones suffice. Start at three and raise it only when you see context missing.

The fifth is using the in memory client in production. A process restart wipes the whole database, and across several instances each holds its own inconsistent set of data.

The sixth is skipping search quality evaluation. Thirty real questions with the expected fragment marked suffice to compare two embedding models and two splitting strategies. Without that set, tuning comes down to impressions.

FAQ

Is Chroma production ready?

Yes, following the Rust core rewrite and the launch of the hosted version. The local variant suits sets up to a few hundred thousand fragments in one process, and at greater scale or across several instances the right choice is the cloud version or another distributed database.

Chroma or Pinecone?

Chroma wins on simplicity of start and on the same code running locally and in the cloud. Pinecone wins on operational maturity under heavy traffic and predictable latency. At the prototype stage, choosing Chroma does not close the door on switching later.

Do I need a separate vector database?

Not always. With a few thousand fragments, in memory search or a PostgreSQL extension suffices. A separate database starts paying off at tens of thousands of entries, metadata filters, and a requirement that queries return within tens of milliseconds.

Can data stay on my infrastructure?

Yes, the open source version runs entirely locally, and embeddings can be computed by a model served through Ollama. In that arrangement no document fragment leaves your infrastructure.

How does Chroma connect to agent frameworks?

It has ready integrations with LangChain and LlamaIndex, where it appears as a vector store behind a standard interface. You can also use it directly, which for simple search is often clearer than an intermediate layer.

Documentation sits on the project site, and the source code in the GitHub repository.