A chunk in your index reads: "The deadline is 30 days from the date of purchase, and proof of original payment is required."
Deadline for what? Which product? Whose policy? The paragraph made perfect sense in the document, three sections below a heading that established all of it. Splitting the document threw that away, and now this chunk is nearly unretrievable — it does not contain the words a user would search for.
This is the failure that no amount of chunk-size tuning fixes, because the problem is not where you cut. It is that cutting removes context the passage was relying on.
What changed in 2026
- Enrichment moved to indexing time. Rather than trying to recover context at query time, systems started adding it when documents are ingested, which is cheaper per query and more reliable.
- Prompt caching made it economical. Contextualising every chunk means sending the document repeatedly; caching the document as a stable prefix cut that cost dramatically.
- Hybrid retrieval became the standard pairing. Contextualised chunks improve both embedding search and keyword search, and combining the two outperformed either.
- The technique spread beyond text. The same idea — annotate the fragment with where it came from — got applied to code, tables, and transcript segments.
What the technique does
For each chunk, generate a short description situating it in its source document, and prepend that to the chunk text before embedding and indexing.
The stored text becomes something like: "This chunk is from the ACME Pro warranty policy, in the section on return eligibility for hardware purchases." followed by the original passage.
Now the chunk contains the vocabulary a user would actually search with — the product name, the topic, the document type — none of which appeared in the original text. A query about "ACME Pro return deadline" matches directly.
|
Plain chunking |
Contextual retrieval |
| Chunk contains topic words |
Only if present in the passage |
Always, via the prepended context |
| Handles pronouns and references |
Poorly |
Well |
| Indexing cost |
Negligible |
One model call per chunk |
| Query cost |
Unchanged |
Unchanged |
| Works with keyword search |
Weakly |
Strongly — adds searchable terms |
| Fixes |
Nothing structural |
The context-loss failure mode |
The row worth noting is indexing versus query cost. This is an up-front investment paid once per chunk, after which every query benefits. That is a much better trade than query-time techniques, which pay on every request forever.
Keeping the indexing cost sane
The obvious objection: generating a description for every chunk means one model call per chunk, and a large corpus has a lot of chunks.
Three things make it manageable.
Cache the document. You send the same document with each of its chunks. That document is a stable prefix, so prompt caching applies directly and cuts the dominant cost. Structure the request so the document comes first and the chunk last.
Use a small model. Producing a one-sentence description of where a passage sits is not a hard task. A cheap fast model does it well.
Batch it. This is offline work with no latency requirement, which is exactly what batch inference endpoints are for — see batch API for LLMs.
Together these usually bring the cost of contextualising a corpus down to something unremarkable, paid once at ingestion.
Pairing it with keyword search
Contextual retrieval improves embedding search and improves keyword search more than people expect, because the prepended text adds exact terms — product names, document titles, section headings — that keyword matching handles far better than embeddings do.
That makes hybrid retrieval the natural pairing: run both an embedding search and a keyword search over the contextualised chunks, then merge the rankings. Each catches what the other misses. Embeddings handle paraphrase and concept; keyword handles exact identifiers, error codes, and proper nouns that embeddings blur together.
The merge step matters. Reciprocal rank fusion is a common approach and needs no tuning, which is a virtue when you would otherwise be hand-weighting two incomparable score scales.
Common mistakes
- Contextualising already self-contained chunks. FAQ entries and standalone definitions gain nothing and cost a call each.
- Generating context without the document. A model asked to describe a chunk in isolation invents the context, which is worse than none.
- Making the context too long. A sentence or two. A paragraph dilutes the chunk's own content in the embedding.
- Not caching the document prefix. The single largest avoidable cost.
- Re-contextualising unchanged chunks on every reindex. Cache by content hash.
- Skipping keyword search. Half the benefit of the added terms goes unused.
FAQ
How is this different from adding metadata?
Metadata sits in structured fields and is used for filtering. Contextual retrieval puts the context into the text that gets embedded, so it influences similarity directly. Both are useful and they do different jobs — filters narrow the candidate set, context improves matching within it.
Does it replace good chunking?
No. Bad chunk boundaries still split sentences and separate a claim from its qualifier. Contextual retrieval fixes lost context, not bad cuts — RAG chunking strategies still applies.
What does it cost to reindex?
The full contextualisation cost again, unless you cache by chunk content. Since most reindexing changes only some documents, caching means you usually pay for the delta.
Does it help with code?
Yes, and the analogue is direct: prepend the file path, the enclosing class or module, and the imports. A function body without that context is as ambiguous as a policy paragraph without its heading.
Where to go next
For measuring whether it improved anything, read retrieval metrics. For the cost mechanism that makes it affordable, prompt caching, and for the query-side technique that attacks the same gap, query rewriting for RAG.