Unstructured, the layer between a file and retrieval

Unstructured, the layer between a file and retrieval

Before a document reaches retrieval it must stop being a file. That conversion looks like a technical step and in practice decides a system's quality more than the choice of vector database or model.

Unstructured handles exactly that stage. It accepts dozens of formats, recognises document structure, and returns a list of elements labelled by type: title, paragraph, table, list item, page header. Fragments for indexing are assembled from those.

Elements instead of text

The key difference from ordinary text extraction lies in what comes out of the tool.

pip install "unstructured[pdf]"

from unstructured.partition.auto import partition

elements = partition(filename="contract.pdf")

for e in elements:
    print(type(e).__name__, e.text[:60])

The result is not a string but a list of objects with types and metadata. You know a given fragment is a title and another a table cell, and you know which page it came from.

That opens possibilities flat text lacks. Headers and footers can be filtered out, since they are recognised as such. Tables can be handled separately, preserving structure. Splitting can follow titles, since their positions are known.

Filtering by type is the simplest and most often skipped optimisation. Removing headers, footers, and page numbers before indexing noticeably improves accuracy, since those repeating elements clutter every fragment and dilute its meaning.

Parsing strategies

The tool offers several modes differing in cost and quality, and choosing between them is the first decision.

The fast mode extracts text from the file's text layer. It runs instantly and costs nothing, while seeing no layout, recognising no tables, and being useless on scans.

The high resolution mode analyses page layout with a model recognising graphical elements. It separates columns, detects tables, and copes with scans, at a cost in time and compute.

The intermediate mode applies high resolution parsing only where needed, meaning pages without a text layer. That is a sensible default on a mixed archive where some files are digital documents and some are scans.

elements = partition(
    filename="report.pdf",
    strategy="hi_res",
    infer_table_structure=True,
    languages=["pol", "eng"],
)

Naming languages matters for optical character recognition and is the most often skipped parameter. The model assumes English by default, so diacritics in other languages come out mangled without it, and the error spreads across the whole archive.

Splitting into fragments

The second stage assembles elements into fragments ready for indexing, and the tool offers several strategies here.

Splitting by title treats a heading as a boundary, so a fragment corresponds to a document section. That is the most sensible choice for documentation, contracts, and reports, since a section is the natural unit of an answer.

Basic splitting joins elements up to a size limit regardless of structure. Simpler and worse, useful on documents without clear headings.

from unstructured.chunking.title import chunk_by_title

chunks = chunk_by_title(
    elements,
    max_characters=1200,
    combine_text_under_n_chars=200,
)

The parameter combining short fragments matters more than it appears to. Without it a document with many short sections shatters into dozens of two sentence fragments, none holding enough context to answer anything.

Review the splitting output before building anything further. Fifteen minutes of reading fragments usually shows the parameters need adjusting, and that changes answer quality more than any prompt tuning.

Open source versus the platform

That distinction deserves understanding, since the vendor's material states it less plainly than it should.

The open source library handles dozens of formats and offers the basic parsing strategies. The vendor states, though, that it is not intended for production use and performs markedly worse at extracting content from documents and tables.

The paid platform adds better layout and text recognition models, content enrichment with vision models, additional chunking strategies, and data source connectors. Billing runs per page: the free plan covers 15,000 pages a month with no card required, and above that the rate is $0.03 per page. The allowance resets monthly, so unused pages do not carry over. The bill stops at $3,000 a month, though, and past that figure every further page is free up to a million pages a month.

The practical conclusion is to test the open source version on your hardest documents before deciding. If your files are plain text or simple documentation, the open source version suffices and costs nothing. If they are scans and tables, the difference is sometimes large enough to decide.

Price it by page count rather than document count. An archive of a thousand hundred page reports is a hundred thousand pages, an entirely different bill from a thousand single page letters. For files without pages, such as email or spreadsheets, the vendor counts a page as 100 KB of incoming data, so the bill there follows size rather than record count.

Source connectors

A separate part of the offering covers connectors pulling documents from where they live and depositing results where they belong.

That solves a problem more common in deployments than parsing quality: a manually updated archive stops being updated. A connector pulling files incrementally from a document store means a new file reaches the index without anybody remembering.

On the output side, vector stores are available, including the options covered in the pieces on Qdrant and Chroma. That means the whole path from file to indexed fragment can be assembled from ready parts.

Note, though, that a connector solves fetching rather than deletion. A document withdrawn from the source must disappear from the index too, and that needs handling deliberately, otherwise answers rest on content that no longer exists.

Unstructured against the alternatives

The last row is an approach that became a real alternative in recent years. Sending a page as an image to a model understanding image and text gives good results on complex layouts with nothing to configure. The price is a cost per page, so on a hundred thousand page archive it drops out, while on two hundred pages it is often simplest.

Choosing between the first two rows depends on whether you want a library or an application. The first plugs into your own pipeline and leaves the rest to you. The second gives a complete set with an interface at the cost of a heavier deployment.

Formats beyond PDF

PDF files draw all the attention, while a company archive in practice consists of a dozen other things, and the tool is often more useful there.

Word processor files carry explicit structure written into the format, so recognising headings and tables is certain here rather than guessed from layout. That is the best possible case: results are exact and parsing is fast and free.

Presentations break into slides, and each slide into a title and bullets. Worth knowing, though, that speaker notes are often more valuable than the slides themselves, since full sentences live there while the slide holds keywords.

Email needs a separate decision. A message with quoted thread history contains the same text repeatedly, and indexing everything means one statement enters the index in ten copies. Stripping the quoted portion before indexing is often the single largest quality improvement on a mail archive.

Locally saved web pages process fine, though menus, footers, and consent banners land in the output alongside the content. Filtering by element type helps, but works less reliably on pages than on documents, since a navigation heading is often recognised as an ordinary title.

Spreadsheets are the boundary case. Formally they process, in practice a table flattened into text rarely answers anything sensibly. If the numbers matter, reading the sheet directly and querying a database beats feeding it into semantic search.

Metadata worth keeping

Every element carries a description of its origin, and that part is the most underrated when building retrieval.

The page number lets you show the user where an answer came from. That looks like a detail and decides trust in the system more than the model's quality does. An answer with a link to a specific page can be verified, an answer without one must be taken on faith.

File name and source path enable filtering search by directory or document type. On an archive spanning contracts, reports, and correspondence that is often the only way to stop a question about contract terms from surfacing mentions in email.

Dates deserve attention too, since documents go stale. A three year old policy and its current version look nearly identical to retrieval, while differing in exactly what matters most to the user.

Carry these fields into the vector database alongside the fragment text. Recovering them later, after indexing the text alone, means reprocessing the archive from scratch.

Performance on a large archive

Processing a few hundred documents is a matter of minutes, processing a few hundred thousand needs a plan.

High resolution mode is slow, since it runs a layout model on every page. On a large set the bottleneck becomes available compute rather than network or disk, so parallelising across processes yields a real gain.

Save results to disk after parsing and before chunking. Chunking is cheap and you will repeat it while tuning parameters, whereas parsing is expensive and there is no reason to repeat it every time. That single habit saves hours during tuning.

Incremental processing matters from the start. Storing a content hash per file lets you skip unchanged ones, turning an overnight run across the whole archive into a few minutes of catching up on the difference.

With an archive that grows daily, processing stops being a script and becomes a pipeline with a schedule, retries, and dependencies between steps. Airflow describes that as a graph of tasks and tracks what ran, what failed, and what is waiting, and on a run interrupted halfway that knowledge is worth more than firing at the right hour. Its third version added event driven triggering, so a new file at the source starts processing immediately instead of waiting for the nightly run.

Plan separately for files that will not process. On a large collection there will always be documents that are corrupt, password protected, or saved in a format variant the library does not handle. Writing them to a list for manual review beats halting the whole run on one file, and beats skipping it silently without a trace.

Common mistakes

The first is skipping the language parameter in optical character recognition. Diacritics then come out mangled, and the error spreads across the archive, surfacing only during retrieval.

The second is fast mode on documents containing scans. It returns empty text or fragments, and since it reports no error, the missing content is noticed only when retrieval finds nothing.

The third is indexing every element, headers and footers included. Repeating text in every fragment dilutes its meaning and lowers accuracy.

The fourth is not combining short fragments. A document with many sections shatters into dozens of two sentence pieces, none holding enough context.

The fifth is estimating cost by document count rather than page count. On reports that difference runs to a hundredfold.

The sixth is skipping the output review. Checking twenty generated fragments takes a quarter of an hour and shows more than any parameter description.

FAQ

What is Unstructured for?

Turning documents into data ready for indexing. It recognises file structure and returns a list of typed elements rather than flat text, so headers can be filtered, tables preserved, and documents split by section.

Is the open source version enough?

For simple, digital documents usually yes. The vendor states it is not intended for production use and copes less well with tables and scans, so on a hard archive compare both variants on your own files.

What does the platform cost?

Billing runs per processed page: 15,000 pages a month are free, above that the rate is $0.03 per page, and the bill stops at $3,000 a month. Unused free pages do not carry into the next month. When estimating, count pages rather than documents, since on reports the difference is manifold.

How does it differ from RAGFlow?

It is a library and platform for document processing that you plug into your own pipeline. RAGFlow is a complete application with its own interface, storage, and search, run on your own infrastructure. The first gives more freedom, the second less work.

How do I check whether parsing works correctly?

Review the generated fragments for ten of your hardest documents. If tables kept their structure, columns do not interleave, and headers do not appear mid sentence, the configuration is sound. That is the only method giving certainty.

Documentation sits on the project site, and the open source code in the GitHub repository.