When a retrieval-augmented system gives a wrong answer, there are two possible causes. Either the retriever did not fetch the right information, or it did and the generator failed to use it properly. Those require completely different fixes, and an end-to-end quality score tells you nothing about which one happened.
Measuring the stages separately is the single most useful thing you can do to a RAG system.
What changed in 2026
- Stage-separated evaluation became standard. Tooling and practice converged on reporting retrieval and generation metrics independently rather than as a combined score.
- Faithfulness got distinguished from correctness. Recognition spread that an answer can be perfectly grounded in a retrieved document that is itself wrong or outdated.
- Real question sets displaced synthetic ones. Questions generated from the documents themselves proved too easy, since they are phrased using the document's own vocabulary.
- Retrieval-only diagnostics matured. Measuring recall at various cutoffs became routine, which made it obvious how often retrieval was the actual bottleneck.
The metrics that matter
| Stage |
Metric |
What it tells you |
| Retrieval |
Recall at k |
Was the needed chunk fetched at all — the ceiling on everything |
| Retrieval |
Precision at k |
How much noise came with it |
| Retrieval |
Mean reciprocal rank |
How high the right chunk ranked |
| Generation |
Faithfulness |
Are claims supported by the retrieved context |
| Generation |
Answer relevance |
Does it address the question asked |
| Generation |
Completeness |
Did it use all the relevant retrieved content |
| End to end |
Correctness |
Is the final answer actually right |
| End to end |
Refusal appropriateness |
Does it decline when context is insufficient |
Recall at your working cutoff is the metric to look at first. It sets a hard ceiling: if the right chunk is not in the retrieved set, no amount of prompt engineering or model upgrading will produce a correct answer. Teams frequently spend weeks on generation quality while recall sits at a level that caps their achievable accuracy.
The faithfulness and correctness distinction is subtle and important. Faithfulness asks whether the answer is supported by what was retrieved. Correctness asks whether it is true. An answer can be perfectly faithful to a document that is out of date, and only correctness catches that. Both are worth measuring because they point at different problems — faithfulness failures are generation problems, correctness failures with high faithfulness are corpus problems.
Building the test set
Use real questions. The tempting shortcut is generating questions from your documents automatically, and it produces a systematically easy test set — the generated question uses the document's own phrasing, so retrieval finds it trivially. Real users ask using different words, which is precisely the hard part.
Label the supporting chunks, not just the answers. To measure retrieval you need to know which chunks should have been fetched for each question, and that labelling is the tedious part with no shortcut.
Include questions your corpus cannot answer. A system that confidently answers everything is worse than one that declines appropriately, and you cannot measure that without unanswerable cases in the set.
Refresh it as user phrasing shifts, which is the query drift described in embedding drift explained.
Common mistakes
- One combined score. Cannot distinguish retrieval failure from generation failure.
- Synthetic questions from the documents. Systematically too easy.
- No unanswerable questions. Cannot measure appropriate refusal.
- Tuning generation when recall is the constraint. Optimizing the wrong stage.
- Faithfulness alone as a quality measure. Faithful to a wrong document is still wrong.
- Never relabelling. The set ages as content and phrasing change.
FAQ
What recall should I target?
High enough that retrieval is not your bottleneck. If recall at your cutoff is well below your target answer accuracy, retrieval is the limiting stage regardless of anything else.
Can a model judge faithfulness reliably?
Reasonably well, since it is a comparatively concrete task — check each claim against provided context. Calibrate against human labels as described in LLM as judge explained.
How large should the evaluation set be?
A hundred labelled questions is a solid working size. Thirty is enough to start and detect large problems.
Do these metrics apply to agentic retrieval?
Yes, with the addition of rounds-per-question as a cost metric. See agentic RAG explained.
Where to go next
For improving the retrieval side, read RAG chunking strategies and hybrid search with BM25. For grounding claims to sources, citation grounding in LLMs.