A user asks your RAG system a question and gets a confident, fluent, wrong answer. The instinct is to blame the model — better prompt, stronger model, tighter instructions.
Frequently the model did nothing wrong. It was handed three chunks of context, none of which contained the answer, and it did what models do with insufficient context: produced something plausible. The failure happened in retrieval, before generation started, and no amount of prompt engineering fixes it.
Measuring the two stages separately is the difference between debugging and guessing.
What changed in 2026
- Retrieval evaluation separated from answer evaluation. Teams stopped treating RAG as one black box to score end-to-end, because end-to-end numbers tell you something is wrong without telling you where.
- Agentic retrieval added a dimension. Systems that search iteratively made "did we find it" a question about a loop rather than a single query, with rounds-per-question becoming its own metric.
- Position effects got taken seriously. The finding that models attend unevenly across long contexts made rank position matter, not just presence.
- Hybrid retrieval became default. Combining vector and keyword search improved recall enough that measuring which component contributed became a practical need.
The metrics worth knowing
| Metric |
Question it answers |
Use when |
| Hit rate @k |
Did the right chunk appear in the top k at all? |
Always — start here |
| Recall @k |
What fraction of relevant chunks did we get? |
Multi-document questions |
| Precision @k |
What fraction of retrieved chunks were relevant? |
Context budget is tight |
| MRR |
How high did the first relevant result rank? |
One correct answer exists |
| NDCG |
Are results ordered by relevance? |
Graded relevance matters |
| Context relevance |
Did the model receive usable material? |
Judged, expensive, holistic |
Hit rate is where to start, and a lot of teams never need to go further. It requires only a set of questions with known correct chunks, and it answers the question that actually matters: is the answer even in the context? If hit rate is 60%, then 40% of your failures are retrieval failures and no prompt change touches them.
Recall over precision at this stage, generally. A missing document is unrecoverable — the model cannot use what it never saw. An irrelevant document is usually just noise the model ignores. The exception is when context is tight or you are paying per token, where precision starts to cost real money.
MRR and NDCG matter more than people expect because of position. Models do not attend uniformly across a long context; material at the beginning and end gets more weight than the middle. A correct chunk at position twelve of fifteen is present and may as well not be. If hit rate is high but answers are still poor, ranking is the likely culprit.
Building the measurement
You need questions paired with the chunks that should answer them. That is the work, and there is no way around it.
The cheapest honest approach: take real user questions, run retrieval, and have a human mark which returned chunks were actually relevant. A few dozen questions is enough to establish a baseline and detect regressions. This doubles as the seed for a golden dataset on the generation side.
The tempting shortcut is generating synthetic questions from your documents — take a chunk, ask a model to write a question it answers, and you have a labelled pair for free. It works for coverage and it is systematically biased: synthetic questions use the document's own vocabulary, which is exactly what makes retrieval easy. Real users phrase things differently, and that gap is where retrieval fails. Use synthetic pairs to bulk out a set, not to establish a baseline.
Once you can measure retrieval alone, the debugging path becomes clear:
- Low hit rate → chunking or embedding problem. See RAG chunking strategies.
- High hit rate, poor ranking → reranking is likely the highest-value addition.
- Good retrieval, bad answers → now it is genuinely a generation problem, and prompt work will help.
Without the split, all three present identically as "the answers are bad."
Common mistakes
- Only measuring end-to-end quality. Tells you something broke, not where.
- Tuning chunk size against answer quality. Too many variables move together to learn anything; tune against retrieval metrics.
- Optimising precision at the retrieval stage. Recall is what you cannot recover later.
- Ignoring rank position. Present-but-buried is closer to absent than to found.
- Synthetic questions only. They share vocabulary with the source and overstate performance.
- Fixed k for every query. Simple lookups need few chunks; synthesis questions need more.
- Not tracking rounds in agentic retrieval. A system answering well in five rounds when two would do is quietly expensive.
FAQ
What k should I retrieve?
Enough that hit rate is acceptable, no more. Larger k improves recall and costs context and money, and it pushes relevant material further down where attention is weaker. Measure hit rate across several k values and find where the curve flattens.
Does a reranker help?
Often substantially, when hit rate is good and ranking is poor — that is precisely the situation it addresses. It adds latency and a second model call, so confirm from your metrics that ranking is the problem before adding one.
How do I measure retrieval for agentic RAG?
Same metrics per round, plus rounds-to-answer as its own number. A system that eventually finds the right material after four searches has different economics from one that finds it in one, even at identical final quality. AI agents vs RAG covers where the loop belongs.
Should I use an LLM to judge retrieval relevance?
It works for scaling up labelling and needs validation against human labels first, like any judge. For hit rate specifically you often do not need one — exact chunk matching against known answers is cheaper and unambiguous.
Where to go next
For the chunking decisions that drive hit rate, read RAG chunking strategies. For the storage layer and its retrieval behaviour, best vector databases for RAG, and for the generation-side measurement this pairs with, LLM evaluation metrics.