A model posts an excellent score on a well-known benchmark. The natural reading is that it is good at what the benchmark measures. An alternative reading is that the benchmark, which has been on the public internet for years, was in the training data.
Both are consistent with the score. Distinguishing them from outside is difficult, which is the whole problem with contamination: it does not produce an error, it produces a number that looks like success.
What changed in 2026
- Contamination became an assumed condition. Rather than a scandal, it turned into a default assumption about any long-published benchmark.
- Private evaluation displaced public leaderboards. Serious model selection moved to task-specific held-out data.
- Held-out benchmark variants appeared. Organisations began maintaining unpublished test splits specifically to measure without leakage.
- Synthetic data got scrutinised. The realisation that model-generated evaluation data can inherit contamination from the generator changed how synthetic sets are built.
How it happens
Direct inclusion. The benchmark's questions and answers appear verbatim in a scraped training corpus. Common for anything hosted on a public repository or discussed in blog posts.
Indirect inclusion. The questions appear in tutorials, papers, forum discussions, or model cards that were themselves scraped. The benchmark need not have been ingested directly for its content to be present.
Derived contamination. Evaluation data generated by a model that itself saw the benchmark. Using a strong model to synthesise test cases can reproduce material it memorised.
Format leakage. Even without exact questions, heavy exposure to a benchmark's format and style inflates performance on it relative to genuine capability.
| Source |
Detectability |
| Verbatim inclusion |
Sometimes, via memorisation probes |
| Indirect discussion |
Very difficult |
| Synthetic derivation |
Difficult |
| Format familiarity |
Nearly impossible |
Signals of contamination
None are conclusive; together they are suggestive.
A score far above related tasks. A model excelling on one benchmark while performing ordinarily on similar problems it has not seen is a strong hint.
Perfect recall of benchmark specifics. If a model can reproduce the exact wording of a question given a fragment, it has seen it.
Performance collapsing on perturbed versions. Rephrase the questions, change the numbers, alter the format. Genuine capability transfers; memorisation does not. This is the most practical test available.
A wide gap between public and private results. If a model tops a public leaderboard and performs ordinarily on your own data, that gap is the information.
Building sets that stay clean
Do not publish them. The moment an evaluation set is public it is a candidate for future training data. This is the single most important discipline, and it conflicts with the desire to share benchmarks — which is why the field has drifted toward private evaluation.
Build from your own data. Real production examples from your system have never been on the public internet. They are also more relevant to your task than any general benchmark — see golden datasets.
Refresh periodically. Even a private set may leak through screenshots, support tickets, or a well-meaning blog post. Rotating in new cases limits the exposure of any one vintage.
Be careful with synthetic generation. If you generate test cases with a model, they may reflect what that model memorised. Human review of a sample, and perturbation of the generated cases, both help.
Perturb deliberately. Keep a variant of your set with rephrased questions and changed specifics. A model that performs well on the original and poorly on the variant has memorised rather than learned.
Common mistakes
- Selecting a model on leaderboard position. Contaminated results, general task, not yours.
- Publishing your evaluation set. Ends its useful life.
- Trusting synthetic data without review. May carry the generator's memorisation.
- Treating a very high score as good news. On a well-known benchmark it warrants suspicion.
- Never perturbing. Removes your best contamination test.
- Never refreshing. Private sets leak slowly.
- Comparing models across different eval vintages. Not comparable if contamination differs.
FAQ
Are public benchmarks useless?
Not useless — they are a rough filter for general capability and a poor basis for a decision. Use them to narrow candidates, then test properly on your own data.
How do I know if my own set is clean?
If you built it from your production data and never published it, it is very likely clean. The main leak routes are screenshots, tickets, and documentation.
What if a vendor claims decontamination?
Reasonable providers do attempt it, and it is genuinely hard — indirect and format contamination resist filtering. Treat it as a good-faith mitigation rather than a guarantee.
Does this affect fine-tuning evaluation?
Yes, and in an additional way: your fine-tuning data may overlap your evaluation data. Splitting before training rather than after is the basic discipline.
Where to go next
For building uncontaminated evaluation, read golden datasets. For the related problem of benchmarks ceasing to discriminate, benchmark saturation, and for evaluation decay generally, eval drift.