A benchmark is useful while it separates models. Once the top several score 94.1, 94.4, and 94.6, it has stopped doing that — the differences are smaller than the measurement noise, and ranking on them is ranking on chance.
That happens well before a benchmark reaches its ceiling, which is why "there is still headroom" is a poor argument for continuing to use one.
What changed in 2026
- Saturation accelerated. Benchmarks that had been discriminating for years compressed within months as models improved.
- Harder benchmarks appeared and saturated too. The cycle of building a harder test and watching it compress became routine.
- Attention shifted to task-specific evaluation. Teams largely stopped selecting models on general leaderboards.
- Label quality got audited. Analysis of residual errors on saturated benchmarks repeatedly found flawed ground truth rather than model failure.
Label errors dominate the residual
The finding that reframes what a saturated benchmark's remaining errors mean.
When models reach the mid-nineties on a widely-used benchmark, a substantial share of the remaining "errors" turn out to be cases where the benchmark's answer is wrong, ambiguous, or defensible either way. Every large public benchmark carries some rate of label noise, and it stops being negligible exactly when model performance approaches it.
Two consequences follow. There is an effective ceiling below 100%, set by the label error rate — a model cannot exceed it without agreeing with wrong answers. And differences near that ceiling are meaningless, because they largely reflect which flawed labels each model happened to match.
A model scoring 96 against another at 95 may be worse and better at matching mistakes.
Saturation and contamination look the same
Both produce high scores that do not reflect capability, and from outside they are difficult to separate.
|
Saturation |
Contamination |
| Cause |
Models genuinely improved |
Test data was in training |
| Scores |
High, clustered |
High, possibly isolated |
| Perturbation test |
Performance holds |
Performance collapses |
| Fix |
New harder benchmark |
Private benchmark |
The perturbation test distinguishes them. Rephrase the questions and change the specifics: a genuinely capable model holds up, a contaminated one falls apart. That test is worth running before drawing any conclusion from a strong benchmark result — see eval contamination.
What to use instead
Your own task data. A set built from your production examples measures the thing you actually care about, does not saturate on anyone else's schedule, and is not contaminated. This is the durable answer — see golden datasets.
Multi-dimensional scoring. Accuracy alone hides cost, latency, and reliability. A model marginally more accurate and twice as expensive is not better for most products, and a single benchmark number cannot express that.
Behavioural properties. Format compliance, refusal calibration, tool-calling reliability, and instruction adherence differentiate models sharply even where accuracy has converged. These are frequently what actually determines whether a model works in your product.
Head-to-head on real requests. Running candidate models on a sample of production traffic and comparing outputs is more informative than any leaderboard, and it is not much work — see A/B testing LLM models.
Common mistakes
- Ranking on a saturated benchmark. Differences are noise.
- Treating the ceiling as 100%. Label noise sets a lower effective maximum.
- Ignoring cost and latency. Accuracy alone is not a decision.
- Assuming a new benchmark is better. Newer benchmarks saturate too, and faster.
- Not perturbing. Cannot distinguish capability from memorisation.
- Reporting a single number. Hides the dimensions that differentiate.
FAQ
When is a benchmark saturated?
When the spread across leading models is comparable to the measurement noise, or to the label error rate. At that point it has stopped ranking anything.
Should I ignore benchmarks entirely?
No — they are a useful coarse filter to exclude clearly weaker models. They are a poor basis for choosing between the leading ones.
How large should my own eval set be?
Smaller than people expect. A few dozen well-chosen cases covering distinct failure modes discriminates better than thousands of generic ones.
What replaces benchmarks for tracking progress?
Task-specific evaluation with your own data, tracked over time, plus behavioural properties. That is genuinely more informative and requires more work, which is why leaderboards persist.
Where to go next
For building your own measurement, read golden datasets. For the related failure of leaked test data, eval contamination, and for comparing candidates on real traffic, A/B testing LLM models.