Someone tweaks a system prompt to fix a formatting complaint. The fix works. It also makes the model more verbose in a different flow, changes how it handles an empty input, and quietly increases how often it refuses an edge case that used to work. None of this surfaces until users find it, because the only verification was checking that the original complaint went away.
This is the ordinary condition of LLM development without regression testing. The prompt is the most behaviour-dense file in the codebase and frequently the least tested.
What changed in 2026
- Prompts entered the review process properly. Treating prompt files as code — version controlled, reviewed, and gated by tests — moved from good practice to normal practice.
- Model upgrades forced the issue. Enough teams got burned by a version bump silently changing output behaviour that pre-upgrade regression runs became routine.
- Tiered suites replaced all-or-nothing runs. Full evaluation on every commit is too slow and expensive; a fast deterministic tier plus a slower judged tier became the standard shape.
- Judge validation became part of the suite. If a model scores your outputs, the judge's agreement with human labels is itself something that can regress.
What to assert on
Exact-match assertions are the instinct from ordinary testing and mostly the wrong tool here. Outputs vary between runs; a test asserting a specific sentence will fail on a rephrasing that is entirely correct.
| Assertion type |
Cost |
Reliability |
Example |
| Schema / format |
Trivial |
Very high |
Valid JSON, required fields present |
| Constraint |
Trivial |
Very high |
Under 200 words, no forbidden phrase |
| Substring / regex |
Trivial |
High for facts |
Contains the correct account number |
| Tool call shape |
Cheap |
High |
Called refund with the right argument |
| Refusal behaviour |
Cheap |
High |
Declines the out-of-scope request |
| Semantic similarity |
Moderate |
Medium |
Response resembles the reference answer |
| Model judgement |
Expensive |
Medium |
"Does this answer the question?" |
The order matters. Deterministic checks at the top are fast, free, and catch a surprising share of real regressions — malformed JSON, a missing citation, a response that doubled in length. Reach for model judgement only for the qualities the cheap checks genuinely cannot express, and remember that a judge is another non-deterministic component you now depend on. LLM-as-judge covers using it responsibly.
Building the suite from real failures
The most valuable test case is one that already broke something in production. It is pre-validated as important, it has a known correct behaviour someone argued about, and it will recur.
Make it a rule with no exceptions: every production LLM bug gets a test case before the fix merges. That single discipline, applied for a few months, produces a suite that reflects how your system actually fails rather than how you imagined it might. Teams that write test cases speculatively end up with suites that cover the failure modes they anticipated, which are by definition not the ones that surprised them.
The same pipeline feeds your evaluation dataset — the cases worth regression-testing and the cases worth including in a golden dataset overlap almost entirely.
Tiering so CI stays usable
A full evaluation suite on every commit is too slow to tolerate and expensive enough to notice on the bill. Three tiers work well.
On every commit — seconds, no model calls. Prompt templates render without error, required variables are present, tool schemas are valid, no prompt exceeds its size budget. Pure static checks that catch a real class of mistake for free.
On prompt or model changes — a minute or two, a small sample. Twenty to fifty representative cases with deterministic assertions. Fast enough to gate a merge, broad enough to catch obvious breakage.
Before release, and on a schedule — the full suite. Everything, including judged cases and expensive scenarios. Slow and thorough, run when the latency is affordable.
Run the middle tier on prompt edits specifically. A change to a .txt prompt file that skips CI because it is "not code" is exactly the change most likely to alter behaviour, and it is a common gap in otherwise disciplined pipelines. The caching patterns in CI/CD caching strategies help keep the middle tier fast enough that people do not route around it.
Common mistakes
- Asserting exact output strings. Produces a flaky suite that gets muted, which is worse than no suite.
- Setting temperature to zero to force determinism. You are now testing a configuration you do not run in production, and outputs still are not perfectly reproducible.
- Judging with the model under test. It shares the blind spots that caused the failure.
- No test for refusals. Prompt changes shift refusal behaviour readily, and an assistant that starts declining valid requests looks fine to any accuracy metric.
- Skipping the pre-upgrade run. Model version bumps change behaviour without erroring. Run the suite against the new version before switching.
- Never removing tests. Cases covering removed features slow the suite and measure nothing.
- Testing only the happy path. Empty input, enormous input, wrong language, injection attempts, and malformed tool results are where real systems break.
FAQ
How do I handle flaky tests?
Distinguish genuine non-determinism from a weak assertion. A test that flakes because the wording changed needs a better assertion. A test that flakes because the model is genuinely inconsistent on that input has found a real reliability problem, and quarantining it hides a bug.
Should I run these against a real model or a mock?
Both, at different tiers. Mocked responses test your parsing, error handling, and control flow deterministically and cheaply. Real calls test whether the model actually behaves. Mocks alone will pass while your prompt is broken.
How many test cases are enough?
Start with your known production failures — usually a dozen or two — and grow only when something escapes. A suite that grows from real escapes stays relevant; one grown to hit a target number fills with cases nobody understands.
What about testing agents rather than single calls?
Same principles, harder execution. Assert on the trajectory — which tools were called, in what order, with what arguments — rather than only the final output, since a correct answer reached by a wrong path will break the next time. AI agent observability covers capturing those traces.
Where to go next
For the production-side counterpart that catches what your suite never anticipated, read online evals for LLM apps. For assembling the cases themselves, golden datasets for LLM evals, and for planning the upgrades that break prompts, AI model deprecation planning.