Cohere Embed, or images and text in one space

Cohere Embed, or images and text in one space

Embed is Cohere's family of embedding models, and the whole family has done something since its third version that most competitors do not offer: images and text land in the same vector space. The fourth iteration additionally lets you supply both in one input.

That sounds like a technical detail and opens a concrete use. A query written in words can find a chart, a screenshot, or a scanned invoice, without describing images in text and without maintaining two separate indexes.

This text covers the embedding model itself: parameters, compression, and traps. A broader account of the vendor's offering, reranking and private deployment included, sits in a separate piece.

Input type, the parameter you must supply

This is the most common source of poor results on first use and the thing setting this model family apart from the competition.

On every call you state what the vector is for. A document entering the index gets marked differently from a user query, and differently again from text meant for classification or clustering.

import cohere

client = cohere.ClientV2()

documents = client.embed(
    texts=chunks,
    model="embed-v4.0",
    input_type="search_document",
    embedding_types=["float"],
)

question = client.embed(
    texts=["What is the notice period?"],
    model="embed-v4.0",
    input_type="search_query",
    embedding_types=["float"],
)

The model treats both cases differently, since a query and a document differ in nature. A query is short and interrogative, a document long and declarative, so embedding them the same way worsens the match.

The most common mistake is indexing a collection as queries or querying with a document vector. Search then works, returns results, and is simply noticeably worse, so the problem often goes unnoticed for months.

Images and compound documents

The second half of the offering concerns content beyond text, and that is this family's main advantage.

The model accepts an image and returns a vector in the same space as text vectors. That means one index serves both, and a text query compares directly against images.

Mixed inputs are accepted too, meaning text interleaved with images within one fragment. That matches what real documents look like: a report page with a paragraph, a chart, and a caption beneath it.

The use where the difference runs largest is documents whose content lives in the visual layout. A slide deck, a form, an invoice, a technical diagram. The classic route requires recognising text from the image, reconstructing the structure, and embedding the result, and each of those steps loses some information. Embedding the page as an image skips that whole chain.

A practical caveat: images are billed separately, at close to four times the text rate per token, and a page sent as an image consumes more of them than its text alone. On an archive where most documents are ordinary text, route only the pages whose text does not convey them to the image path.

Dimension truncation and quantisation

Here the model offers more options than most competitors, and on large collections these settings decide the vector database bill.

The first axis is dimension count. By default a vector holds one thousand five hundred and thirty six values, and a shorter one can be requested, down to two hundred and fifty six. The model was trained so the most important information sits at the start of the list, so truncation does not destroy the result.

The second axis is number format, and it gets skipped despite a larger impact on cost. Instead of floating point values you can request single byte integers or a binary form where each dimension occupies one bit.

response = client.embed(
    texts=chunks,
    model="embed-v4.0",
    input_type="search_document",
    embedding_types=["int8"],
    output_dimension=1024,
)

The arithmetic here is persuasive. A thousand dimension vector in floating point takes four kilobytes, the same one as single bytes takes a kilobyte, and in binary form a hundred and twenty eight bytes. Across ten million fragments that is the difference between forty gigabytes and slightly over one.

The accuracy loss is markedly smaller than that proportion suggests, while the binary form usually requires a second pass: fast search over the compressed form, then refinement against full vectors. Vector databases such as Qdrant and Weaviate support that arrangement directly.

Test your settings on your own data rather than from a table. Fifty real questions with expected answers, the same collection indexed three ways, and a comparison of how often the right document landed in the top five.

Long context and chunking

The model accepts very long inputs, on the order of a hundred thousand tokens in one call. That tempts you to embed whole documents instead of splitting them, and it is usually a bad idea.

The reason is simple: a vector averages the whole content. A document spanning twelve topics produces a vector sitting close to none of them, so retrieval stops landing.

Long inputs help with something else: embedding fragments that are naturally long, a whole contract chapter or a report page together with its chart. You then avoid splitting something that forms a whole merely because the model has a short limit.

The practical rule stays the same regardless of vendor. A fragment should match the unit in which an answer to a question fits. For documentation that is a section, for a contract a clause, for a report a page. The model's limit is an upper bound here rather than a guideline.

Bear the billing in mind too. Sending a hundred thousand token document costs exactly as much as sending a hundred fragments of a thousand tokens, because you pay per token rather than per call. There is therefore no saving in embedding the whole thing, only worse relevance, which makes that choice a loss on both sides.

There is one case, though, where long input saves work: documents that cannot sensibly be split automatically. A table spread across three pages, or a form with linked fields, loses meaning once cut, so it is better to embed it whole and accept worse relevance than to split it at a point that destroys the meaning. Before calling a document unsplittable, though, check the tools that read page layout, such as Unstructured: they recognise a table, a heading and a list as distinct elements and cut along their boundaries rather than by character count, which removes some of these cases entirely.

Image search in practice

Since this is the model's main advantage, the whole pipeline deserves showing, along with where the decisions hide.

The first concerns what you actually embed. For a slide deck the natural unit is a slide, for a report a page, for a product catalogue a single photo together with its description. The choice of unit matters more here than with text, since an image cannot be split as easily as a paragraph.

The second concerns resolution. An image at too low a resolution loses fine text on a chart, while too high a one costs more and gives nothing back. Check across a few real pages at what resolution the model starts answering questions about chart contents correctly, and adopt that as the standard.

The third concerns what to do with the result. Search returns a page or a slide, and the user wants to see where exactly the answer sits. Keeping the page number, file name, and a thumbnail alongside the vector lets you show the source, and that decides trust in the system more than raw accuracy does.

The fourth, most often skipped, concerns mixed collections. An archive with a thousand text documents and a hundred decks deserves two track processing: text through the cheap text path, decks through the image path. The vectors land in one index, since they share a space, and the bill runs several times lower than treating everything as images.

Measuring retrieval quality

Without a set of cases every settings change is guesswork, and this model carries plenty of settings, so building that set right away pays off.

Start with fifty real questions, ideally ones people actually ask rather than ones invented for a test. Attach to each the document or fragment holding the answer. That work takes two hours and repays itself on the first parameter change.

Measure two things. The first is the share of questions where the right document landed in the top five results, since that is usually how many reach the context of the model generating the answer. The second is the average position of the right document, since it shows the direction of a change even when the first measure holds still.

The same set answers four questions at once: does dimension truncation hurt, does quantisation hurt, does a different chunking approach help, and is the image path worth paying for. Without it each of those ends in a discussion built on impressions.

Record the baseline result before any optimisation too. Without a reference point it is hard to establish whether three weeks of tuning produced anything beyond fatigue.

Cohere Embed against the alternatives

The decision reduces to one question: does your collection hold content that text does not convey. If the archive is notes, documentation, and correspondence, the cheaper text option suffices and you will not notice a difference. If it is decks, forms, and scans, image handling changes retrieval quality enough to justify the higher price.

The last row deserves deliberate consideration, since it gets chosen by momentum. Two separate models mean two indexes, two similarity scales, and a need to merge results from both, a harder task than it looks, since values from different spaces are not directly comparable.

Common mistakes

The first is omitting the input type or using the same one for documents and queries. Search works, returns results, and is noticeably worse, so the problem goes unnoticed for a long time.

The second is embedding whole documents because the model accepts long inputs. A vector averages content, so a document spanning many topics stops matching anything.

The third is routing every page through the image path. Images cost close to four times more per token and give nothing beyond the text version on ordinary text.

The fourth is skipping quantisation on a large collection. The difference in database footprint reaches orders of magnitude at an accuracy loss measured in percentage points.

The fifth is mixing vectors from different models or different settings in one index. They are not comparable, so results turn random.

The sixth is picking settings from a table rather than from measurement. A set of fifty real questions with expected answers settles it in an afternoon.

The seventh is indexing a collection without recording the settings it was built with. Six months on nobody remembers whether vectors were computed at a thousand dimensions or at the full width, and without that, adding new documents to an existing index produces entries incomparable with the rest.

FAQ

How does Embed v4 differ from the previous version?

In mixed inputs, context length, and the choice of dimension count. The earlier version accepted images too, but every input had to be of one kind, the limit was five hundred and twelve tokens, and the vector held a fixed one thousand and twenty four values. The fourth accepts text, images, and text interleaved with images in one input, holds a hundred and twenty eight thousand tokens, and lets you choose the vector length.

What should the input type parameter be set to?

For documents entering the index, the value meant for documents; for user queries, the value meant for queries. Separate values exist for classification and clustering too. This parameter is required and omitting or confusing it noticeably worsens accuracy.

Does quantisation ruin the results?

Less than the saving proportion suggests. The single byte form gives a four times smaller index at a loss measured in percentage points, and the binary form many times smaller, while that last one usually needs a refinement pass against full vectors.

What does it cost against cheaper options?

Text costs six times more here than with the cheapest providers, and images close to four times more than text. On an ordinary text archive the difference is often unjustified; on documents with visual layout, retrieval quality justifies it.

Where should the resulting vectors live?

In any vector database supporting your chosen number format. With quantisation, check that before deciding, since not every database supports the binary form with refinement, and adding it later means rebuilding the index. On a small collection the option covered in the piece on pgvector suffices.

The models are documented on the vendor's site, and the fourth version's details in its release note.