Retrieval-augmented generation is sold as the fix for hallucination: ground the model in real documents and it will stop making things up. In production, that promise only holds if every stage of the pipeline — chunking, embedding, retrieval, and generation — is actually working. Each stage fails quietly, in ways that look like the system is functioning normally right up until someone checks an answer against the source and finds it is wrong.
What changed in 2026
- Evaluation moved from ad hoc to standard practice. Teams that shipped RAG on vibes in 2023–2024 now run retrieval-precision and answer-faithfulness evals as part of CI, because manual spot-checking did not scale.
- Long-context models raised the bar for "is RAG even needed." With context windows large enough to hold entire document sets, some teams now compare RAG against a stuffed-context baseline before building a retrieval pipeline at all.
- Hybrid retrieval became the default, not the exception. Pure vector similarity search is now usually paired with keyword or BM25 search and a reranking step, because vector-only retrieval misses exact-match queries (part numbers, names, error codes) too often.
- Reranking models got cheap enough to run on every query, closing much of the gap between "retrieved something relevant" and "retrieved the actually useful chunk."
The pitfalls, in the order they usually get discovered
Chunking that ignores document structure. Splitting purely by character or token count cuts tables in half, separates a heading from its content, and breaks numbered steps across chunks. The retriever then returns fragments that read as plausible but are missing the context needed to answer correctly.
Embedding model mismatch. Using a general-purpose embedding model on a domain with specialized vocabulary (legal, medical, internal jargon) produces embeddings that cluster documents by surface similarity rather than actual relevance. Domain-tuned or at least domain-evaluated embeddings matter more than teams expect.
Retrieval-generation mismatch. The retriever returns five reasonable chunks; the model picks the wrong one to answer from, or blends details across chunks incorrectly. This is a generation-side failure that retrieval quality alone cannot fix — it requires prompting the model to cite which chunk it used and checking that.
Stale indexes. A RAG system is only as current as its last index update. If source documents change and the vector store is not re-embedded, the system answers confidently from outdated information with no signal to the user that anything is wrong.
No negative case handling. When nothing relevant is in the index, a well-built system says so. A poorly built one retrieves the closest-matching chunks anyway — because similarity search always returns something — and the model generates an answer from irrelevant context.
Failure modes at a glance
| Pitfall |
Symptom |
Primary fix |
| Bad chunking |
Answers missing context that exists in the source |
Structure-aware chunking (headings, tables, sections) |
| Embedding mismatch |
Retrieval returns topically-near but wrong documents |
Evaluate embeddings on your actual queries, not benchmarks |
| Retrieval-generation mismatch |
Right chunks retrieved, wrong one used in the answer |
Require citation of source chunk; verify against it |
| Stale index |
Confident answers from outdated data |
Scheduled re-indexing tied to source update events |
| No negative case handling |
Answers generated from irrelevant chunks |
Similarity threshold + explicit "not found" path |
How to actually catch these before users do
Build a small evaluation set of real questions with known-correct answers and known-correct source chunks, and run it against every pipeline change. Track two numbers separately: retrieval precision (did the right chunk come back) and answer faithfulness (did the model use it correctly). Conflating the two hides which stage is actually broken — the same discipline behind AI safety evals applies here: score the pipeline continuously, not just once before launch.
FAQ
Does adding a reranker fix most retrieval problems?
It helps significantly with borderline relevance ranking, but it cannot fix chunks that are structurally broken or an index that is missing the right document entirely. Reranking improves ordering among what was retrieved; it does not fix what was never indexed correctly.
Is RAG still worth building given how large context windows have gotten?
For a small, mostly-static document set, stuffing full context is often simpler and can outperform a poorly tuned RAG pipeline. RAG earns its complexity at scale — large or frequently changing corpora where context windows cannot fit everything.
How often should the index be refreshed?
It depends entirely on how often the source data changes. Tie re-indexing to actual update events where possible rather than a fixed schedule, so staleness is bounded by design rather than by a calendar guess.
Can RAG eliminate hallucination completely?
No. It substantially reduces ungrounded fabrication when retrieval works, but the model can still misread, misattribute, or embellish details from correctly retrieved content.
Where to go next