The standard way teams build an AI feature is to write a prompt, try five examples by hand, decide it looks good, and ship. Then it breaks on the sixth kind of input nobody tried, someone edits the prompt to fix it, and that edit silently breaks the first five. There is no test suite, so nobody notices for two weeks.
Eval-driven development is the obvious correction: build the measurement first, then build the thing.
What changed in 2026
- Prompts entered version control properly. Treating prompt text as a reviewed, diffed, tested artifact rather than a config string became normal engineering practice rather than a maturity signal.
- Evals moved into continuous integration. Running a subset of the evaluation suite on every pull request that touches a prompt became affordable once small judges and deterministic checks carried most of the load.
- Production failures became the eval source. The best-performing teams stopped writing synthetic test cases and started promoting real failures into the suite, which keeps the set grounded in what actually goes wrong.
- Segmented reporting replaced single scores. A headline number proved actively misleading, and dashboards shifted to per-category breakdowns.
What to put in the suite
| Check type |
Runs without a model |
Catches |
| Schema and format validation |
Yes |
Broken JSON, missing fields, wrong types |
| Forbidden content patterns |
Yes |
Policy violations, leaked internals, banned phrasing |
| Exact or fuzzy reference match |
Yes |
Regressions on questions with known answers |
| Retrieval grounding check |
Partly |
Claims not supported by the provided context |
| Judged quality comparison |
No |
Tone, completeness, helpfulness |
| Adversarial and injection cases |
Partly |
Prompt injection, jailbreak drift after a prompt edit |
Weight the top of that table heavily. Deterministic checks are fast, free, and catch a surprising share of real regressions — malformed output, dropped fields, a rewritten prompt that stopped enforcing the format. Judged checks are the expensive layer and should cover only what deterministic checks cannot express. Getting the judge right is its own discipline, covered in LLM as judge explained.
Building the set from real failures
Start small and grounded. Take your last thirty production complaints or bad outputs, write down the input, the wrong output, and what correct would have looked like. That is your first evaluation set, and it is more valuable than a synthetic suite ten times the size, because every case is a mistake your system actually made.
Then keep it growing by rule: every production bug adds a case before the fix ships. This is ordinary regression-test discipline applied to a probabilistic system, and it works for the same reason it works in normal software.
Segment from the beginning. Tag each case with its task type — extraction, summarization, refusal, multi-turn — and report per segment. Aggregate scores hide the failure mode where a prompt change improves the common case and destroys an uncommon but important one.
Watch for contamination as the set ages. If you tune prompts against the same hundred cases repeatedly, you eventually fit the set rather than the task. Hold out a portion you never look at during iteration, and check it only before release.
Common mistakes
- Building evals after the feature ships. By then the prompt is tuned to whatever the author tried informally, and the eval set gets written to pass.
- All-synthetic test cases. They test the failure modes you imagined, not the ones users find.
- One aggregate number. It moves for reasons you cannot diagnose and hides category-level regressions.
- No held-out split. Iterating against the full set overfits it, and the score stops predicting production behavior.
- Running evals only manually. If it is not in CI, it will be skipped on the change that needed it most.
FAQ
How big should an eval set be?
Start at thirty real cases. Grow toward a few hundred as the feature matters more. Size matters less than whether the cases represent real inputs.
How much does running evals in CI cost?
Less than expected if deterministic checks run on every commit and judged checks run only on prompt-touching changes or nightly. That split keeps the common path nearly free.
Should I evaluate the retrieval separately from the generation?
Yes. A wrong answer caused by bad retrieval needs a different fix than one caused by bad generation, and a combined score cannot tell you which.
What about non-deterministic output making tests flaky?
Set temperature low for evaluation runs, score properties rather than exact strings where possible, and accept a pass band rather than a single threshold.
Where to go next
For the metrics themselves, read LLM evaluation metrics. For scoring subjective quality at scale, LLM as judge explained, and for catching what evals miss once you are live, AI agent observability.