Sentence Transformers, or vectors without leaving your machine

Sentence Transformers, or vectors without leaving your machine

Sentence Transformers is a library computing embeddings locally, on your own hardware, without calls to external services. It began in academia; today the team behind the largest collection of open models develops it, and the project moved under their repository.

The alternative to paid services is real here rather than merely theoretical. Open embedding models are good enough today that on company document search the gap against commercial providers is often small, while per token cost disappears entirely.

The price is hardware, time spent choosing a model, and upkeep. Below I describe when that arithmetic comes out positive, since the answer is not obvious and depends on scale.

Three kinds of model

The library supports three model classes with different purposes, and distinguishing them pays off, since online material often blends them.

A dense model turns text into a vector of a few hundred numbers, each meaning something you cannot name. That is the classic embedding familiar from commercial services and the starting point for most uses.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("intfloat/multilingual-e5-large")

vectors = model.encode_document(chunks)
query = model.encode_query("What is the notice period?")

Separating document embedding from query embedding, visible in those two methods, matters more than it looks. Some models expect a prefix before the text, one for a document and another for a question, and omitting it worsens accuracy in a way invisible without measurement. These methods add it themselves, but only when the model declares the prefixes in its configuration file for this library. The model in the example above does not declare them, so the document and query markers have to be passed by hand there, through the parameter that takes a prefix. Before assuming it happens automatically, check the model card for whether the prefixes are recorded there.

A sparse model returns a vector with an enormous number of dimensions, almost all of them zeros. That sounds odd and carries a concrete point: the dimensions correspond to words, so such a vector combines meaning based search with exact word matching. It solves a problem that with ordinary embeddings requires building hybrid search from two separate indexes.

A pair scoring model judges the fit between a query and a document while seeing both at once. It gives markedly better results than comparing vectors, at a cost in time, since it must run for each pair separately. Hence the two stage arrangement: cheap vector search returns fifty candidates, and this model picks the best five.

That third kind is the most underrated in practice. Adding it to existing search is usually a dozen or so lines of code and the largest single jump in accuracy achievable without changing the embedding model or how documents are split. Try it before fine tuning, since it costs a fraction of that work.

Choosing a model

This is the decision where losing a week is easiest, so approaching it methodically pays off.

Start with language. Models trained on English alone perform poorly on documents in other languages, even when they score highly in benchmarks. A multilingual model is the starting point here rather than a compromise.

Then size. A model of a few hundred million parameters runs on an ordinary processor at sensible speed, while one several times larger needs a graphics card to process an archive in reasonable time. The accuracy gap is often smaller than the size gap suggests.

Finally, measurement on your own data. Take fifty real questions with expected answers, index the same collection with three models, and compare how often the right document landed in the top five. That work takes an afternoon and settles more than a week of reading benchmarks.

Treat public leaderboards as a candidate list rather than an answer. They measure tasks averaged across many domains, and your documents belong to one specific domain.

Fine tuning on your own data

Here lies an advantage commercial services do not offer at all: the ability to teach the model your domain.

The gain can be large with specialist vocabulary. A general model does not know that in your company two different terms mean the same thing, or that an abbreviation used in the documentation refers to a particular product. Fine tuning on a few thousand question and answer pairs from your archive teaches it those links.

from sentence_transformers import SentenceTransformerTrainer, losses

loss = losses.MultipleNegativesRankingLoss(model)

trainer = SentenceTransformerTrainer(
    model=model,
    train_dataset=training_set,
    loss=loss,
)
trainer.train()

The simplest and usually sufficient form of training data is pairs: a question and the fragment holding the answer. You need no negative examples, since the loss function treats the other pairs in a batch as negatives.

Where to get those pairs is a harder question than the training itself. Three sources work in practice: search query history together with what users clicked, support tickets together with their answers, and pairs generated by a language model from your documents and reviewed by a person.

A caveat worth stating: fine tuning makes sense only once you measure that a general model falls short. It is several days of work, while improving chunking or adding a candidate scoring step usually gives more for less.

Performance and hardware

The arithmetic for local embeddings differs from a service, since the cost is fixed rather than variable.

A processor suffices for queries arriving one at a time. A mid sized model computes a vector for a short query in tens of milliseconds, which in a typical application disappears into the response time of everything else.

A graphics card becomes necessary for indexing. Processing a hundred thousand fragments on a processor takes hours, on a card minutes. If you do it once, hours are acceptable. If the archive updates daily, a card repays itself quickly.

Three settings make the biggest difference without changing model. The first is batch processing rather than one at a time, since a model computes many texts together far more cheaply than in sequence. The second is lower numeric precision, which roughly halves the time at a practically imperceptible loss. The third is sorting fragments by length before processing, so each batch holds texts of similar size.

Mind memory too. A loaded model occupies it for the process's whole lifetime, so loading it inside a function that starts on every request is expensive and slow. Load once at application start and keep it.

Running it in production

The library computes vectors and its role ends there, so the rest of the system is yours. A few things deserve planning before a model reaches production.

The first is how the model is exposed. Loading it in every application process means as many copies in memory as you have processes, and with a model occupying two gigabytes that stops fitting quickly. A more sensible arrangement is a separate service computing vectors, queried by the rest of the application, with one copy of the model in memory.

The second is warm up. The first call after loading takes noticeably longer than later ones, since buffers allocate and computation paths compile only then. Running one synthetic request at startup removes that spike, which otherwise hits the first user after every deployment.

The third is versioning. Record alongside the index the model name, its version, and the settings under which the vectors were produced. Without that, adding documents after a library update or after pulling a newer model version yields entries incomparable with the rest, and the symptom is simply worse results, with no error.

The fourth is model download at startup. By default the library fetches weights from the network on first use, which in an environment without internet access or during sudden scaling is a problem. Bake the model into the container image or mount it from a volume, and startup becomes predictable.

What it really costs

Comparing against a paid service is worth doing on numbers, since intuition misleads in both directions here.

On the service side the bill is simple: token count times rate. Indexing a million fragments of five hundred tokens is five hundred million tokens, meaning anywhere from ten to several dozen dollars once, plus pennies for queries.

On the self hosted side the cost has three parts, two of which get skipped in such comparisons. The first is the machine, running from tens to hundreds of dollars a month depending on whether you need a graphics card. The second is the time to choose a model and deploy it, meaning several days of work. The third is upkeep: updates, monitoring, and reacting when something stops working.

From that follows a simple threshold. For a one off indexing of a small archive the service wins outright. At steady high volume self hosting starts winning, since the cost stops growing with traffic. With data that cannot leave the company the arithmetic is irrelevant, since there is simply no alternative.

Price it once, on your own numbers, rather than relying on a belief that local is always cheaper. In a typical application with a few thousand queries a day, the embedding bill with a provider is often lower than the cost of one machine.

Sentence Transformers against the alternatives

The first row wins in two situations. The first is data that cannot leave the company, and then there is nothing to discuss. The second is high steady volume, where per token cost accumulates month after month while a machine costs the same regardless of traffic.

It loses on a prototype and at small scale. Standing a model up, choosing it, and maintaining it is work that across a thousand documents costs more than a few dollars of API calls.

Common mistakes

The first is omitting the prefixes a model requires. Some models expect different markers for a document and a query, and the methods intended for each add them automatically only when the model has them recorded in its configuration. With a model that does not declare them, you must supply them yourself.

The second is loading the model on every request. That takes seconds and occupies memory, while loading once at process start suffices.

The third is processing fragments one at a time. Batches are many times faster and it is one of the few optimisations that costs nothing.

The fourth is picking a model from a leaderboard without testing on your own data. Benchmarks measure averaged tasks, and your documents belong to one domain.

The fifth is fine tuning before measuring whether a general model suffices. Improving chunking usually gives more for a fraction of the effort.

The sixth is mixing vectors from different models, or different versions of the same model, in one index. They are not comparable and search then returns arbitrary results.

The seventh is downloading model weights from the network at service start. In an environment without internet access, or on a sudden increase in instance count, that ends in a failed start at the worst possible moment.

FAQ

Do local models match paid services?

On text document search the gap is often small, particularly after fine tuning on your own data. On tasks requiring image handling or the highest accuracy in a narrow domain, commercial services still lead. Measurement against your own question set settles it.

What hardware is needed?

A processor suffices for serving queries, since a single query computes in tens of milliseconds. A graphics card helps with indexing large collections, where it turns hours into minutes, and repays itself quickly on an archive updated daily.

How does a pair scoring model differ from an ordinary one?

An ordinary model computes a document's vector once, without knowing the question. A pair scoring model sees the query and the document together, so it judges the fit more precisely at a cost in time. Hence the two stage arrangement: fast vector search, then scoring a few dozen candidates.

When is fine tuning worthwhile?

Once you measure that a general model cannot handle your vocabulary and that improving chunking plus adding candidate scoring did not suffice. You need a few thousand question and fragment pairs, ideally from your own search history or support tickets.

Where should computed vectors live?

In any vector database, such as the one covered in the piece on Qdrant. The library computes vectors and its role ends there, so storage and search belong to a separate layer you choose independently.

Documentation sits on the project site, and the code in the GitHub repository.