BGE, or open embeddings with three modes at once

BGE, or open embeddings with three modes at once

BGE is a family of open embedding models developed by a Chinese research institute and released under a permissive licence allowing commercial use. You run the models yourself, so there is no per token cost and no data leaving your infrastructure.

The best known model in this family carries a differentiator competitors usually lack: one model returns three different representations of the same text in a single pass. An ordinary dense vector, a sparse vector corresponding to words, and a set of vectors describing individual parts of the text.

That solves a problem which with other models requires maintaining two separate search systems. Below I describe what follows from that in practice and when it is worth using.

Three modes in one model

Understanding how those representations differ pays off, since it determines which you will use.

A dense vector is the classic embedding: a few hundred numbers where proximity means similar meaning. It handles a question phrased differently from the answer very well and misses exact matches.

A sparse vector holds an enormous number of dimensions corresponding to words, almost all of them zeros. It behaves like keyword search, except the model judges which words matter and accounts for related forms. It lands where a specific number, code, or proper name counts.

The third mode returns a separate vector per part of the text and compares them against the query's parts. It gives the highest accuracy and costs the most in space and time, so it is usually applied as a final step, refining a few dozen candidates.

from FlagEmbedding import BGEM3FlagModel

model = BGEM3FlagModel("BAAI/bge-m3", use_fp16=True)

result = model.encode(
    chunks,
    return_dense=True,
    return_sparse=True,
    return_colbert_vecs=False,
)

The key point is that all three arise in one pass. With other options, hybrid search requires two models, two indexes, and two passes over the archive. Here you pay once and decide what to use.

Multilingual support that works

That is the second differentiator, and on documents in languages other than English it matters more than the first.

The model supports over a hundred languages and, more importantly, handles retrieval between them. A question in one language can find a document in another, since texts with the same meaning across languages sit close together in the same space.

On a company archive that can be decisive. Technical documentation in English, correspondence in the local language, contracts in both: one index serves everything, with nothing translated and no per language collections.

Do verify that on your own data, though, since quality differs between languages. The model was trained on a corpus where some languages are represented better than others, so declared support for a hundred languages does not mean equal accuracy across all of them.

The test takes an afternoon: fifty real questions in your language with expected answers, a comparison against a commercial model on the same collection, and a check of how often the right document landed in the top five.

The model also accepts exceptionally long input, around eight thousand tokens, roughly twenty pages of text. That helps with documents which cannot sensibly be split, a table spread across several pages for instance. It is not, however, an invitation to embed whole documents, because a vector averages the content, so a document covering twelve topics stops matching anything. Treat the model's limit as an upper bound rather than a guideline.

Candidate scoring models

A separate part of the family covers models comparing a query and document pair while seeing both at once.

The difference from embeddings is fundamental. An embedding model computes a document's vector once, without knowing the question, so comparison reduces to the distance between two points. A scoring model sees both together, judging the fit far more precisely at a cost in time, since it must run for each pair separately.

Hence the two stage arrangement, worth treating as the default in serious search. Cheap vector search returns fifty candidates from the whole archive, and the scoring model picks the best five.

Adding that step is usually the largest single jump in accuracy achievable without changing the embedding model or the chunking approach. Try it before fine tuning, since it costs a fraction of that work.

The family includes scoring models in several sizes, variants designed for lower compute cost among them. Choosing between them is an ordinary trade off: larger is more accurate and slower, and at fifty candidates per query that difference translates directly into response time.

Hardware and performance

Models in this family are larger than typical commercial embeddings, since they support many languages and several modes at once, so hardware deserves planning.

For queries arriving one at a time a processor suffices, though compute time is noticeably higher than with smaller models. In an application where the query waits on a language model anyway, that difference disappears entirely.

For indexing, a graphics card changes the situation radically. Processing a hundred thousand fragments on a processor takes many hours; on a card, a quarter of an hour. If the archive updates daily, a card repays itself in the first month.

Three settings make the biggest difference without changing model. The first is reduced numeric precision, roughly halving the time at a practically imperceptible loss. The second is batch processing rather than one at a time. The third is disabling modes you do not use, since computing all three representations costs more than computing one.

That last one gets skipped, and the difference is real. If you are building dense search alone, disable the other two modes in the call and you gain noticeably in time and in database footprint.

Think through where the model should run, too. Loading it separately in every application process means as many copies in memory as you have processes, and these models are large. A more sensible arrangement is a separate service computing vectors, queried by the rest of the application, with one copy of the model loaded once at startup and one synthetic warm up query, so the first real user does not pay for initialisation.

How to merge results from two modes

Since the model gives a dense and a sparse vector at once, you must still decide how to combine both searches into one list. That step decides everything in hybrid search and often gets done by feel.

The naive approach adds the scores together. It does not work, since the scales are incomparable: dense vector similarity usually falls within a narrow range, while a sparse search score can take arbitrarily large values depending on query length.

The approach that works in practice ignores the values and looks only at positions. A document that came third in one search and seventh in the other receives a combined score built from the reciprocals of both positions. The method is simple, needs no threshold tuning, and is robust to one search returning values on an entirely different scale from the other.

The alternative normalises both lists to a common range and sums them with weights. It gives more control and requires tuning those weights against your own question set, so reach for it only once the simpler method proves insufficient.

Verify on your own data whether hybrid search improves the result at all. On descriptive documents, without numbers and codes, dense search alone is often just as good, and adding a second layer only complicates the system.

Fine tuning and the limits of this approach

Models in this family can be fine tuned on your own data, an advantage commercial services do not offer at all.

The gain appears with specialist vocabulary: abbreviations used inside one company, product names, industry terms a general model does not associate with each other. A few thousand question and fragment pairs suffice for the model to learn those links.

The order of operations deserves stating plainly, though, since fine tuning tempts more than it merits. First improve how documents are split into fragments, since that changes accuracy most and costs least. Then add a candidate scoring model. Only when both steps fall short should you reach for fine tuning, since it is several days of work and requires building a training set.

The limit of the whole approach lies elsewhere and deserves knowing. Embeddings find documents similar in meaning to a question rather than answering it. If an answer requires combining information from three documents or computing something, no embedding model will do that, since it is a task for the layer generating the answer.

The second limit is freshness. Vectors reflect a document's state at indexing time, so a changed or withdrawn document must be reprocessed or removed from the index. Without that, search returns content that no longer exists, and that error is harder to notice than an absence of results.

BGE against the alternatives

The second row is not competition here but a layer alongside. The library for working with local models can load models from this family and offers a more convenient interface, so a typical deployment uses both together.

Choosing between the first and third rows comes down to two questions. Whether the data may leave the company, since if not there is nothing to discuss. And whether the volume is large enough for a fixed cost to beat a variable one, which at a few thousand queries a day it usually is not.

Common mistakes

The first is computing all three representations when you use one. Disabling unneeded modes in the call shortens the time and reduces database footprint.

The second is assuming equal quality across every supported language. Declared support for a hundred languages does not mean equal accuracy, so verify your case against your own questions.

The third is skipping the candidate scoring model. That is usually the largest accuracy jump for the smallest amount of work.

The fourth is indexing a large archive on a processor. A graphics card turns many hours into a quarter of an hour and repays itself quickly under regular updates.

The fifth is mixing vectors from different model versions in one index. They are not comparable, and the symptom is worse results with no error message at all.

The sixth is choosing a model by its leaderboard position. Benchmarks measure averaged tasks, and your documents belong to one domain and one language.

FAQ

Is BGE free?

Yes, the models are open and released under a licence permitting commercial use. You pay only for the hardware you run them on and for the time needed to deploy and maintain them.

What does supporting three modes at once mean?

One model returns, in a single pass, a dense vector, a sparse vector corresponding to words, and a set of vectors describing parts of the text. With other options, hybrid search requires two models and two indexes; here one pass over the archive suffices.

Does it work well in languages other than English?

The model supports over a hundred languages and handles retrieval between them, so a question in one language can find a document in another. Quality differs between languages, though, so verify against your own question set before deciding.

What hardware is needed?

A processor suffices for serving individual queries. For indexing a larger archive, a graphics card cuts the time from many hours to a quarter of an hour, making it practically necessary under regular updates.

Where should computed vectors live?

In a vector database supporting your chosen modes, such as the one covered in the piece on Qdrant. With sparse vectors, check support before deciding, since not every database handles them equally well, and adding that later means rebuilding the index.

The models and documentation sit on the project site, and the tooling code in the GitHub repository.