{"post":{"slug":"query-rewriting-for-rag-2026","title":"Query Rewriting for RAG: Fixing the Question First","excerpt":"Users do not phrase questions the way documents phrase answers. Rewriting the query before searching closes that gap and costs one cheap model call.","category":"ai","date":"August 11, 2026","author":"ByteLedger Team","content":"<span class=\"dropcap\">A</span> user types \"why did my card get declined\". Your documentation has a thorough page covering this, titled \"Authorization failure codes and remediation\". The embedding search finds it somewhere around position nine, below several less relevant pages that happen to use the word \"card\" more often.\n\nThe retrieval is not broken. The user and the document are describing the same thing in different vocabularies, and similarity search compares text to text. Query rewriting sits between them: transform what the user asked into something closer to how the answer is written, then search with that.\n\n## What changed in 2026\n\n- **Rewriting moved inside the retrieval loop.** Rather than one transformation up front, systems now rewrite between rounds based on what the previous round actually returned.\n- **Multi-query fan-out became standard.** Generating several phrasings and merging results proved more reliable than trying to produce one optimal query.\n- **Cheap models got good enough.** Rewriting is a transformation task, and small fast models handle it well, which removed the latency objection.\n- **Conversational rewriting became mandatory.** As chat interfaces replaced search boxes, resolving pronouns and references against history stopped being optional.\n\n## The techniques worth knowing\n\n| Technique | What it does | Best for |\n| --- | --- | --- |\n| Conversational rewrite | Resolves references against history | Any chat interface |\n| Multi-query expansion | Generates several phrasings | General recall improvement |\n| Step-back | Asks a broader question first | Narrow questions needing context |\n| Hypothetical answer | Writes a fake answer, embeds that | Vocabulary gap between question and answer |\n| Decomposition | Splits a compound question | Multi-part questions |\n| Metadata extraction | Pulls filters out of the text | Corpora with structured fields |\n\n**Conversational rewriting** is the one nobody can skip. A user asks about pricing, then asks \"does that include tax?\" — that second query is meaningless in isolation. Every embedding of it retrieves generic tax content. Rewriting it to \"does the Pro plan price include tax\" makes it searchable. This single transformation fixes more real failures than everything else combined in chat products.\n\n**Hypothetical answer generation** is the counterintuitive one. Rather than embedding the question, ask a model to write a plausible answer and embed *that*. The generated answer is wrong on facts and right on vocabulary and structure — and since documents are written like answers, it matches them better than a question does. It directly attacks the vocabulary gap.\n\n**Decomposition** handles compound questions. \"How do refunds work and what is the deadline\" is two retrievals pretending to be one, and a single search finds material for whichever half dominates the embedding.\n\n## Fan out rather than perfect\n\nThe instinct is to write one excellent rewritten query. In practice, generating three or four and retrieving for each works better.\n\nDifferent phrasings surface different documents. One uses the user's words, one uses formal terminology, one uses a hypothetical answer. Union the results, deduplicate, and you have better coverage than any single query achieves — and recall is what you cannot recover downstream, per [retrieval metrics](/blog/retrieval-metrics-explained-2026).\n\nThe cost is more retrieval calls, which are usually cheap relative to generation. The thing to watch is context: three queries returning five chunks each is fifteen chunks, many redundant. Deduplicate before assembling the prompt, and rerank if you have one.\n\nExtracting metadata filters during rewriting is worth doing if your corpus has structured fields. \"Show me refund policy changes from last year\" contains a date filter that vector search handles badly and a `WHERE` clause handles perfectly. Pull it out, filter, then search the remainder.\n\n## Keeping it cheap\n\nRewriting adds a model call before retrieval, which adds latency to every query. Three things keep that acceptable.\n\n**Use a small model.** This is transformation, not reasoning. A fast cheap model does it well, and the quality difference against a frontier model is negligible for this task.\n\n**Cache aggressively.** Common questions get rewritten identically every time. A cache on the rewrite step is trivial and removes the latency entirely for repeat queries.\n\n**Skip it when it will not help.** Not every query needs rewriting. A short standalone question in domain vocabulary is already fine. A cheap classifier — or simply checking whether the query contains pronouns or is under a few words — routes around the step for queries that do not need it.\n\n## Common mistakes\n\n- **Rewriting away the user's intent.** An over-eager rewrite can change the question. Keep the original in the retrieval set as a hedge.\n- **Not handling conversation history.** The single largest source of failure in chat RAG.\n- **Using a frontier model for it.** Pays for reasoning you are not using.\n- **Fanning out without deduplication.** Fills context with near-identical chunks.\n- **Rewriting after retrieval.** The point is to improve the search, so it has to happen before.\n- **No measurement.** Compare retrieval hit rate with and without; sometimes it does nothing for your corpus.\n\n## FAQ\n\n**Does this replace a reranker?**\nNo, they operate at different stages. Rewriting improves what gets retrieved; reranking improves the ordering of what came back. Rewriting fixes low hit rate, reranking fixes bad ordering — measure which you have before adding either.\n\n**How many query variants should I generate?**\nThree is a common sweet spot. Beyond that, returns diminish and context fills with redundancy. Measure hit rate against variant count on your own corpus.\n\n**Does it help with long-context approaches?**\nLess, since stuffing more into context reduces retrieval precision pressure — see [long context vs RAG](/blog/long-context-vs-rag-2026). It still helps when the corpus exceeds any window.\n\n**What about non-English queries?**\nRewriting can normalise across languages, which is genuinely useful when your corpus is one language and your users are not. Verify the rewrite model handles your languages before relying on it.\n\n## Where to go next\n\nFor measuring whether rewriting helped, read [retrieval metrics](/blog/retrieval-metrics-explained-2026). For the chunking decisions upstream of it, [RAG chunking strategies](/blog/rag-chunking-strategies-2026), and for the retrieval technique that attacks the same gap from the indexing side, [contextual retrieval](/blog/contextual-retrieval-explained-2026).","hue":202,"metaTitle":"Query Rewriting for RAG: Fix the Question First","metaDescription":"How query rewriting improves RAG retrieval, why follow-up questions break search, and when generating multiple queries beats perfecting a single one.","keywords":["query rewriting","rag query expansion","multi query retrieval","hyde retrieval","conversational search rewriting","retrieval vocabulary gap","query transformation llm","rag optimization 2026"],"tldr":["**The vocabulary gap is the core problem** — users write \"why is my card declined\" and the document says \"authorization failure reasons\".","**Follow-up questions are the clearest win** — \"what about the second one\" is unsearchable until you resolve it against the conversation.","**Generating several queries beats perfecting one** — fan out, retrieve for each, then merge and deduplicate.","**A small model is enough** — rewriting is a cheap transformation, not a reasoning task worth frontier pricing.","**Skip:** rewriting when your users already search in domain vocabulary. Internal tools often need none of this."],"readTime":"5 min read","wordCount":1012},"related":[{"slug":"query-expansion-rag-2026","title":"Query Expansion: Fixing Retrieval Before It Searches","excerpt":"Users ask short, ambiguous questions that retrieve badly. Query expansion rewrites them first. Here is what works, what adds latency for nothing, and when to skip it.","category":"ai","date":"July 25, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":210},{"slug":"agentic-rag-explained-2026","title":"Agentic RAG Explained: Retrieval That Decides","excerpt":"Classic RAG retrieves once and hopes. Agentic RAG lets the model decide what to search, when to search again, and when it has enough. Here is how it works and what it costs.","category":"ai","date":"August 12, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":208},{"slug":"hyde-retrieval-explained-2026","title":"HyDE Retrieval: Searching With an Imagined Answer","excerpt":"HyDE generates a fake answer and searches with that instead of the question. Here is why the trick works, when it beats ordinary retrieval, and its failure mode.","category":"ai","date":"July 25, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":214}],"sameCategory":[{"slug":"agent-cost-attribution-2026","title":"Agent Cost Attribution: Finding Where the Tokens Went","excerpt":"A single monthly API bill tells you that agents are expensive. It does not tell you which feature, which customer, or which retry loop is responsible. That gap is fixable.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":220},{"slug":"human-in-the-loop-agents-2026","title":"Human-in-the-Loop Agents: Where to Put the Approval Gate","excerpt":"Approve everything and the agent is a slow autocomplete. Approve nothing and you have handed irreversible actions to a system that can be talked into things.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":197},{"slug":"reasoning-effort-controls-2026","title":"Reasoning Effort Controls: Tuning How Hard a Model Thinks","excerpt":"Fixed thinking budgets gave way to effort levels, and the default is rarely the right setting. Here is how the dial behaves and where each level actually belongs.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":210},{"slug":"retrieval-metrics-explained-2026","title":"Retrieval Metrics: Measuring the Half of RAG That Fails","excerpt":"When a RAG system answers badly, the retrieval usually failed before the model ever spoke. Measuring generation quality alone hides that entirely.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":215}],"newer":{"slug":"opentelemetry-setup-guide-2026","title":"OpenTelemetry Setup: Instrumenting Without Regret","excerpt":"OpenTelemetry decouples instrumentation from your observability vendor. Here is a setup order that works, what to instrument first, and the cardinality trap.","category":"code","date":"August 11, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":260},"older":{"slug":"reference-check-guide-2026","title":"Reference Checks: What Former Employers Actually Say","excerpt":"Most employers now confirm only dates and title, which changes what a reference is for. Here is how to manage yours.","category":"guides","date":"August 11, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":285}}