Standard dense retrieval compresses an entire document into one vector. That is efficient and lossy — a long document covering several topics gets averaged into a single point that represents none of them precisely. A cross-encoder avoids the loss by processing query and document together, and it is far too slow to run over a whole corpus.
Late interaction is the middle path. It keeps a separate embedding per token, and computes similarity between the query's tokens and the document's at search time.
What changed in 2026
- Storage optimization matured. Compression and pooling techniques cut the multi-vector storage overhead substantially, which had been the main adoption barrier.
- Vector database support broadened. Native multi-vector indexing appeared in more systems rather than requiring a specialized engine.
- Visual document retrieval adopted it. Applying late interaction to page-image patches became a leading approach for document understanding, connecting to multimodal RAG explained.
- The simpler alternative held ground. For many corpora, single-vector retrieval plus a cross-encoder reranker remained competitive and easier to operate.
Where it sits
| Approach |
Quality |
Query speed |
Storage |
Complexity |
| Sparse keyword search |
Good on exact terms |
Very fast |
Low |
Low |
| Single-vector dense |
Good on meaning |
Very fast |
Low |
Low |
| Late interaction |
Better |
Fast |
High |
Medium |
| Cross-encoder over all documents |
Best |
Impractically slow |
None extra |
Low |
| Dense retrieval plus cross-encoder rerank |
Very good |
Fast then moderate |
Low |
Medium |
The last two rows are the real comparison. Late interaction and retrieve-then-rerank both aim to get near cross-encoder quality at practical speed, by different routes. Late interaction pays in storage and gets quality directly at retrieval time. Rerank pays in latency on a small candidate set and keeps storage low.
Which is better depends on your constraint. If storage is cheap and you need low latency, late interaction. If storage is expensive or your corpus is enormous, retrieve-then-rerank as described in cross-encoder reranking.
How the matching works
For each query token, find the most similar token in the document and take that similarity. Sum those maxima across query tokens to score the document.
The effect is that each part of the query gets matched against the best-matching part of the document independently, rather than everything being compressed into one comparison. A query asking about two distinct things can match a document that addresses them in separate sections, which a single averaged vector handles poorly.
This is also why it works well on long and multi-topic documents. Single-vector embedding of a long document produces something that represents its average, and a query about one specific section matches that average weakly. Late interaction matches the section.
The storage consequence follows directly: instead of one vector per chunk you store one per token, which is a large multiple. Compression techniques reduce this considerably and it remains the defining cost.
Common mistakes
- Adopting it before trying rerank. Retrieve-then-rerank is simpler and often sufficient.
- Underestimating storage. The multiple is large; calculate it for your corpus before committing.
- Using it on short documents. The benefit comes from documents a single vector cannot represent well.
- Skipping compression. Uncompressed multi-vector storage is unnecessarily expensive.
- Assuming any vector database supports it. Multi-vector indexing is a specific capability; check.
FAQ
How much more storage does it need?
Substantially more than single-vector — a large multiple depending on document length and compression. Estimate on a sample of your corpus rather than a general figure.
Is it faster than reranking?
At query time, generally yes, because scoring is a similarity computation rather than a model forward pass. The cost moved to storage and indexing.
Does it work with hybrid search?
Yes, as the dense channel in a hybrid setup, fused with keyword results as usual.
Is it worth it for a small corpus?
Rarely. With a small corpus you can afford to rerank generously, which is simpler.
Where to go next
For the alternative path, read cross-encoder reranking. For the sparse counterpart, sparse embeddings explained, and for the visual application, multimodal RAG explained.