Every evaluation number you have ever quoted traces back to a set of examples where somebody decided what the right answer was. The model scored 94%. Against what? Assembled how? By whom, using what standard? Those questions are unglamorous enough that they usually go unasked, which is precisely why so many eval suites quietly measure nothing.
A golden dataset is that foundation: inputs paired with outputs you are willing to defend as correct. Get it right and everything downstream — regression detection, model comparison, prompt iteration — becomes trustworthy. Get it wrong and you have built an elaborate apparatus for generating confident numbers about nothing in particular.
What changed in 2026
- Small curated sets displaced large scraped ones. The field converged on a finding practitioners had suspected for a while: a few dozen examples chosen to cover distinct failure modes outperform thousands sampled at random for the purpose of catching regressions.
- Contamination became a default assumption. Public benchmarks leak into training data. Anything you did not build privately should be treated as potentially memorised, and a suspiciously strong score on a well-known set is now read as a warning rather than a result.
- Multi-dimensional scoring replaced single accuracy numbers. A system that is accurate but spends 26,000 tokens per query is not production-viable, and a scorecard that reports only accuracy will happily recommend it.
- Provenance became a first-class field. Where an example came from, who verified it, and when, are now treated as part of the record rather than tribal knowledge.
Sizing it honestly
| Set size |
Good for |
Not good for |
| 20–50 |
Fast iteration, catching obvious breaks |
Statistical claims, small deltas |
| 100–300 |
Regression suites, model comparison |
Rare edge cases, per-segment analysis |
| 1,000+ |
Fine-grained analysis, subgroup breakdowns |
Anything you must hand-verify frequently |
Most teams need the middle row and reach for the bottom one. The cost that gets ignored is maintenance: every example is something a human must re-verify when your product's definition of correct shifts, and a thousand-example set nobody has re-read in a year is a liability dressed as rigour.
Start at the top of the table. Grow when a real failure slips through, and let each escape add the case that would have caught it. A set that grows this way stays relevant by construction.
Where the good examples come from
Production failures, first and foremost. Anything a user complained about, anything that got escalated, anything you hotfixed. These are pre-validated as important — someone already cared — and they are the cases most likely to recur.
Edge cases you have actually seen. Empty inputs, absurdly long inputs, mixed languages, adversarial phrasing, the malformed request one integration sends every Tuesday. Real ones, not imagined ones; invented edge cases tend to reflect what you expect to break rather than what does.
Cases where reasonable people disagreed. If two reviewers scored an output differently, that example is teaching you something about your own specification, and resolving it improves the spec as much as the dataset.
A deliberate slice of the boring middle. A set consisting entirely of hard cases will mislead you in the other direction, reporting failure on a system that handles normal traffic perfectly well.
What to avoid: synthetic examples generated wholesale by a model, then accepted without review. The failure is circular — you are encoding one model's idea of correctness and then measuring models against it. Generated candidates reviewed by a human are fine and genuinely save time. Generated answers accepted on trust are not.
Keeping it honest over time
Two disciplines separate a dataset that stays useful from one that quietly rots.
Freeze a holdout. Split the set and never look at one half while iterating. The moment you tune a prompt against an example, that example stops measuring capability and starts measuring fit. Teams that skip this see steadily improving scores alongside steadily unchanged user complaints.
Write down the reasoning. Not just the expected output — why it is expected. "Returns null because the account is closed, not because the lookup failed" is a note that lets a future maintainer distinguish a genuine regression from a spec change. Without it, six months later nobody can tell whether a newly-failing example represents a broken model or an outdated expectation, and the usual resolution is to delete the example.
That second habit is what makes a dataset survive contact with a changing product. The scoring side of this — what to measure once you have the examples — is covered in LLM evaluation metrics.
Common mistakes
- Optimising against the whole set. Without a holdout you are measuring memorisation of your own test cases.
- All hard cases. Produces pessimistic numbers and hides regressions on ordinary traffic.
- Unexplained expected outputs. Unmaintainable the moment the original author moves on.
- Never retiring anything. Examples testing removed features inflate your suite and your runtime while measuring nothing.
- One reviewer, no second opinion. Single-annotator sets encode one person's interpretation as ground truth, including the parts that were wrong.
- Ignoring cost and latency. A model that wins on accuracy and loses on both is not a winner; score all three or you will keep recommending the expensive one.
- Treating a public benchmark as your golden set. It measures general capability, is probably contaminated, and says little about your specific task.
FAQ
How often should it change?
Add when something escapes to production. Review the whole set roughly quarterly, or whenever the product's definition of a good answer moves. Rewriting it wholesale more often than that usually means the specification, not the dataset, is unstable.
Can an LLM judge the outputs instead of a human?
For scoring at scale, yes, with care — that is what LLM-as-judge is for. But the judge needs its own validation against human labels, so you still need a human-verified core. It moves the labour rather than removing it.
What about tasks with no single right answer?
Score properties instead of exact matches: does it cite a real source, stay under the length limit, refuse when it should, avoid the forbidden claim. Rubric scoring is less satisfying than exact match and far more honest about open-ended work.
Should the dataset be version-controlled?
Yes, in the repo, reviewed like code. A changed expected output is a meaningful change and deserves the same scrutiny as a changed function.
Where to go next
Once you have the examples, what is an AI eval harness covers running them, and online evals for LLM apps covers the production-side measurement that offline sets cannot give you. For the wider practice, the AI evals guide is the broader starting point.