Retrieval-augmented generation pairs a language model with an external knowledge source it can search at answer time. Instead of relying purely on what was baked into its weights during training, the model retrieves relevant passages from a document store and conditions its answer on them. The pattern is simple in concept and genuinely difficult to execute well — most of the engineering effort lives in the retrieval half, not the generation half.
What changed in 2026
- Hybrid search (keyword plus vector) became the default, not vector-only. Pure semantic search reliably misses exact-match queries — product codes, error strings, names — that keyword search catches trivially.
- Reranking moved from optional to standard. A cheap first-pass retriever now routinely returns 50-100 candidates that a cross-encoder reranker narrows to the top 5-10 actually passed to the model.
- Chunk size guidance shifted toward smaller, more structured chunks with explicit metadata (section headers, source, recency) rather than large fixed-size text blocks.
- Agentic retrieval spread, where the model itself decides whether to search, reformulates the query, and can issue multiple retrieval calls before answering — rather than a single fixed retrieve-then-generate step.
The core pipeline
A standard RAG system has four stages: ingest, retrieve, rerank, generate.
- Ingest — documents are split into chunks, embedded, and stored in a vector index (often alongside a keyword index for hybrid search).
- Retrieve — the user query is embedded and matched against the index, returning a candidate set of chunks.
- Rerank — a more expensive but more accurate model reorders candidates by true relevance to the query.
- Generate — the top-ranked chunks are inserted into the prompt, and the LLM generates an answer grounded in them.
Each stage introduces a place where quality can quietly leak out — a bad chunk boundary in ingestion, a mismatched embedding space, a reranker that was never tuned to your domain.
Chunking strategy
Chunk size is a tradeoff. Too large, and irrelevant text dilutes the signal the model needs; too small, and you lose surrounding context that makes a passage interpretable. Most production systems land somewhere in the range of a few hundred tokens per chunk, with overlap between adjacent chunks so information near a boundary is not orphaned.
Structure-aware chunking — splitting on headings, paragraphs, or semantic boundaries rather than a fixed character count — consistently outperforms naive fixed-size splitting, particularly on structured documents like technical documentation or contracts.
RAG design choices compared
| Design choice |
Option A |
Option B |
When B wins |
| Search type |
Vector only |
Hybrid (vector + keyword) |
Queries include exact terms, codes, or names |
| Chunking |
Fixed-size |
Structure-aware |
Source documents have clear headings/sections |
| Ranking |
Single retrieval pass |
Retrieve + rerank |
Precision at top-k matters more than latency |
| Retrieval trigger |
Always retrieve |
Model decides when to retrieve |
Mixed factual/conversational query load |
| Context window use |
Stuff everything found |
Filter to top few by relevance |
Long contexts degrade attention to buried facts |
Where RAG breaks down
RAG reduces hallucination but does not eliminate it — a model can still misread retrieved context, blend it incorrectly with parametric knowledge, or answer confidently when retrieval returned nothing useful. It also does not fix a fundamentally poor knowledge base: garbage documents retrieved accurately are still garbage. And RAG is the wrong tool for teaching a model a new skill or style; that is a job for fine-tuning, not retrieval.
Latency is the other real cost. Each additional retrieval or rerank stage adds time before the model can start generating, which matters for interactive products.
FAQ
Is RAG better than fine-tuning?
They solve different problems. RAG is for grounding answers in current or proprietary information; fine-tuning is for changing behavior, tone, or task format. Many production systems combine both.
How do I choose a vector store?
It depends on scale, latency needs, and whether you need hybrid search built in. Compare options directly rather than guessing — see the linked vector store comparison below.
Does RAG stop hallucination completely?
No. It substantially reduces hallucination on questions the knowledge base actually covers, but the model can still misinterpret retrieved text or hallucinate when retrieval returns weak or no results.
How many chunks should I retrieve per query?
There is no universal number. Start small (3-5 after reranking) and measure whether answer quality improves as you add more — more context is not always better once attention gets diluted.
Where to go next