A standard retrieval pipeline converts a PDF to text and embeds the text. That works well for a memo and badly for almost everything else a business actually stores. A quarterly report is mostly charts. A lab result is a table whose meaning lives in its layout. A scanned invoice is an image that optical character recognition renders as a soup of disconnected strings.
Multimodal retrieval addresses this by embedding what the page looks like rather than only what text could be pulled out of it.
What changed in 2026
- Page-image retrieval became practical. Rather than parsing a document into text and then embedding, models that embed rendered page images directly moved from research into usable tooling.
- The parsing step got recognized as the bottleneck. Teams measuring retrieval failures traced most of them to extraction rather than to the embedding model or the vector store.
- Hybrid architectures settled as the default. Keeping a text index for precise keyword matching alongside a visual index for layout-heavy content outperformed either alone.
- Cost realism arrived. Early enthusiasm met the reality that page-level visual embeddings produce far more vectors and far larger retrieved payloads than text chunks.
Where text-only pipelines lose information
| Content type |
What text extraction produces |
What is lost |
| Bar or line chart |
Axis labels and a caption, if any |
Every actual value and the trend |
| Complex table |
Cell contents in reading order |
Row and column relationships |
| Scanned document |
OCR text with errors |
Layout, stamps, handwriting, signatures |
| Form |
Field labels and values, often unpaired |
Which value belongs to which field |
| Diagram or flowchart |
Node labels |
The connections that carry the meaning |
| Slide deck |
Bullet fragments |
Visual hierarchy and grouping |
The table row is the one that causes the most silent damage in production. Extraction generally preserves the cell text but flattens the structure, so the model receives a list of numbers with no reliable way to know which row and column each belongs to. It then answers confidently using the wrong cell, which is worse than failing to find anything.
Building a pipeline that works
Start by classifying documents rather than treating the corpus uniformly. Clean text documents go through the ordinary path — parse, chunk, embed — because that is cheaper and gives better keyword precision. Layout-heavy documents get visual treatment.
For the visual path, embed page images and retrieve whole pages, then pass the retrieved page images to a vision-capable model at generation time. This skips parsing entirely, which removes the largest source of error, and it means the generating model sees the chart rather than a description of it.
Budget carefully. A page image embedding scheme can produce many vectors per page, and a retrieved page costs far more context than a text chunk. That interacts directly with the degradation described in context rot — retrieving five full pages fills a window fast. Retrieve fewer, larger units and rely on the model reading them properly.
Keep the text index alongside. Users search for exact identifiers, product codes, and names, and dense visual retrieval is poor at exact matching. The strategies in RAG chunking strategies still apply on that side of the hybrid.
Common mistakes
- Applying visual retrieval to the whole corpus. Most corpora contain a mix, and clean text does not need it.
- Dropping the text index. Exact-match queries degrade badly without it.
- Retrieving too many pages. Visual retrieval units are large; five pages can exceed what the model handles reliably.
- Not measuring extraction quality first. If your parser is losing tables, fixing the parser may be cheaper than rebuilding the pipeline.
- Assuming OCR is solved. It is good on clean print and still unreliable on scans, handwriting, and unusual layouts.
FAQ
Do I need a vision model for generation too?
If you retrieve page images, yes — the generating model must be able to read them. Retrieving images and passing text descriptions defeats the purpose.
Is this more accurate than a good parser?
For layout-heavy content, generally yes, because it removes an entire lossy step. For clean text, a parser is faster and cheaper with no accuracy penalty.
How much more does it cost?
Substantially more in storage, and more per query in context tokens. Measure on a representative sample before committing to it corpus-wide.
Can I use it for handwriting?
Vision models handle handwriting better than traditional OCR, with accuracy that still varies widely by legibility. Test on your actual documents.
Where to go next
For the foundations, read what is RAG and RAG chunking strategies. For the models doing the reading, vision language models explained, and for extraction accuracy specifically, AI document extraction accuracy.