A user types "why did my card get declined". Your documentation has a thorough page covering this, titled "Authorization failure codes and remediation". The embedding search finds it somewhere around position nine, below several less relevant pages that happen to use the word "card" more often.
The retrieval is not broken. The user and the document are describing the same thing in different vocabularies, and similarity search compares text to text. Query rewriting sits between them: transform what the user asked into something closer to how the answer is written, then search with that.
What changed in 2026
- Rewriting moved inside the retrieval loop. Rather than one transformation up front, systems now rewrite between rounds based on what the previous round actually returned.
- Multi-query fan-out became standard. Generating several phrasings and merging results proved more reliable than trying to produce one optimal query.
- Cheap models got good enough. Rewriting is a transformation task, and small fast models handle it well, which removed the latency objection.
- Conversational rewriting became mandatory. As chat interfaces replaced search boxes, resolving pronouns and references against history stopped being optional.
The techniques worth knowing
| Technique |
What it does |
Best for |
| Conversational rewrite |
Resolves references against history |
Any chat interface |
| Multi-query expansion |
Generates several phrasings |
General recall improvement |
| Step-back |
Asks a broader question first |
Narrow questions needing context |
| Hypothetical answer |
Writes a fake answer, embeds that |
Vocabulary gap between question and answer |
| Decomposition |
Splits a compound question |
Multi-part questions |
| Metadata extraction |
Pulls filters out of the text |
Corpora with structured fields |
Conversational rewriting is the one nobody can skip. A user asks about pricing, then asks "does that include tax?" — that second query is meaningless in isolation. Every embedding of it retrieves generic tax content. Rewriting it to "does the Pro plan price include tax" makes it searchable. This single transformation fixes more real failures than everything else combined in chat products.
Hypothetical answer generation is the counterintuitive one. Rather than embedding the question, ask a model to write a plausible answer and embed that. The generated answer is wrong on facts and right on vocabulary and structure — and since documents are written like answers, it matches them better than a question does. It directly attacks the vocabulary gap.
Decomposition handles compound questions. "How do refunds work and what is the deadline" is two retrievals pretending to be one, and a single search finds material for whichever half dominates the embedding.
Fan out rather than perfect
The instinct is to write one excellent rewritten query. In practice, generating three or four and retrieving for each works better.
Different phrasings surface different documents. One uses the user's words, one uses formal terminology, one uses a hypothetical answer. Union the results, deduplicate, and you have better coverage than any single query achieves — and recall is what you cannot recover downstream, per retrieval metrics.
The cost is more retrieval calls, which are usually cheap relative to generation. The thing to watch is context: three queries returning five chunks each is fifteen chunks, many redundant. Deduplicate before assembling the prompt, and rerank if you have one.
Extracting metadata filters during rewriting is worth doing if your corpus has structured fields. "Show me refund policy changes from last year" contains a date filter that vector search handles badly and a WHERE clause handles perfectly. Pull it out, filter, then search the remainder.
Keeping it cheap
Rewriting adds a model call before retrieval, which adds latency to every query. Three things keep that acceptable.
Use a small model. This is transformation, not reasoning. A fast cheap model does it well, and the quality difference against a frontier model is negligible for this task.
Cache aggressively. Common questions get rewritten identically every time. A cache on the rewrite step is trivial and removes the latency entirely for repeat queries.
Skip it when it will not help. Not every query needs rewriting. A short standalone question in domain vocabulary is already fine. A cheap classifier — or simply checking whether the query contains pronouns or is under a few words — routes around the step for queries that do not need it.
Common mistakes
- Rewriting away the user's intent. An over-eager rewrite can change the question. Keep the original in the retrieval set as a hedge.
- Not handling conversation history. The single largest source of failure in chat RAG.
- Using a frontier model for it. Pays for reasoning you are not using.
- Fanning out without deduplication. Fills context with near-identical chunks.
- Rewriting after retrieval. The point is to improve the search, so it has to happen before.
- No measurement. Compare retrieval hit rate with and without; sometimes it does nothing for your corpus.
FAQ
Does this replace a reranker?
No, they operate at different stages. Rewriting improves what gets retrieved; reranking improves the ordering of what came back. Rewriting fixes low hit rate, reranking fixes bad ordering — measure which you have before adding either.
How many query variants should I generate?
Three is a common sweet spot. Beyond that, returns diminish and context fills with redundancy. Measure hit rate against variant count on your own corpus.
Does it help with long-context approaches?
Less, since stuffing more into context reduces retrieval precision pressure — see long context vs RAG. It still helps when the corpus exceeds any window.
What about non-English queries?
Rewriting can normalise across languages, which is genuinely useful when your corpus is one language and your users are not. Verify the rewrite model handles your languages before relying on it.
Where to go next
For measuring whether rewriting helped, read retrieval metrics. For the chunking decisions upstream of it, RAG chunking strategies, and for the retrieval technique that attacks the same gap from the indexing side, contextual retrieval.