Your offline evaluation says the new model is better. It scored higher on three hundred cases you selected, which is genuine information about those three hundred cases.
Real traffic is different. It contains phrasings you did not anticipate, edge cases you did not imagine, and a distribution that has drifted since you built the set. A live comparison measures the model on that, which is ultimately the only question that matters.
What changed in 2026
- Model turnover made this routine. Frequent releases meant teams were comparing candidates continuously rather than annually.
- Shadow deployment became the standard first step. Running a candidate on copied traffic before exposing users became expected practice.
- Behavioural metrics displaced explicit feedback. Retry rates and edit distance proved far more informative than thumbs ratings.
- Cost entered the comparison properly. Quality-per-dollar became a reported metric rather than an afterthought.
Route by user, not by request
The single most common design error is splitting per request.
A user whose first message goes to model A and second to model B gets a conversation with inconsistent behaviour, style, and capability. That degrades their experience in a way neither model is responsible for, and it contaminates every conversational metric you were trying to measure.
Hash a stable identifier — user, session, or account — and route consistently. Everyone gets one model for the duration of whatever unit you are measuring.
The same applies to agent runs: a run that switches models mid-loop produces behaviour attributable to neither.
Shadow first
Before splitting live traffic, run the candidate on copies of real requests without showing anyone the output.
That gives you the candidate's behaviour on genuine traffic distribution — including the long tail your test set lacks — with zero user risk. You can compare outputs, measure latency and cost, and check for errors and refusals before anyone is affected.
Shadow traffic costs real money, since you are paying for inference nobody consumes, so sample rather than mirroring everything. A modest percentage of traffic is usually enough to find the problems.
What shadow cannot measure is user behaviour, since nobody sees the output. That is what the live split is for, and it is why shadow is a first step rather than a substitute.
Metrics that work
| Metric |
Signal quality |
Notes |
| Retry / rephrase rate |
Strong |
User telling you it failed |
| Edit distance before use |
Strong |
Where output is a draft |
| Abandonment |
Moderate |
Noisy individually |
| Copy / export events |
Moderate |
Positive signal |
| Explicit rating |
Weak |
Sparse and biased |
| Judge model score |
Moderate |
Needs its own validation |
| Latency, cost |
Exact |
Always compare these |
Behavioural signals are abundant and honest, which is why they beat asking. A user who immediately rephrases the same question has told you the first answer failed, without being asked and without bias.
Pair them with cost and latency always. A model that scores marginally better and costs three times as much is a worse choice for most products, and quality-only comparisons systematically favour the expensive option.
Reading the results
Two failure modes in interpretation.
Stopping early on a favourable result. LLM output quality is noisy, and short experiments produce apparent differences that do not survive. Decide the duration in advance.
Aggregating over segments. An overall wash frequently hides one segment substantially better and another substantially worse. Break down by request type, user tier, and language before concluding there is no difference — the interesting result is usually in the segments.
Also watch for behavioural changes that are not quality changes: refusal rate, response length, and format compliance can shift between models in ways that affect the product without affecting any accuracy metric — see refusal training.
Common mistakes
- Per-request routing. Inconsistent experience, contaminated metrics.
- Skipping shadow. Users find the problems first.
- Quality without cost. Systematically favours expensive models.
- Stopping early. Noise mistaken for signal.
- Aggregate-only analysis. Hides segment reversals.
- Not re-tuning the prompt for the candidate. Prompts are model-specific; comparing an untuned candidate is unfair to it.
- Skipping offline evaluation. Live traffic is an expensive way to learn what a test set would have shown.
FAQ
How long should a comparison run?
Long enough to cover your traffic's natural cycles — weekday and weekend patterns at minimum — and to accumulate enough events for your noisiest metric. Decide before starting.
Should I re-tune the prompt for the new model?
Yes, and it complicates the comparison. An untuned candidate is being judged on someone else's prompt. The honest approach is to tune both and compare tuned versions, and to be explicit that you did.
What if results are mixed?
Common, and usually informative rather than inconclusive. Better on some segments and worse on others suggests routing by request type rather than picking one globally.
Does this replace offline evaluation?
No — offline evaluation gates the deploy and catches regressions on known cases. Live comparison measures real-world performance. Both, in that order.
Where to go next
For the offline foundation, read golden datasets and LLM regression testing. For tracking which model is deployed where, AI model registry, and for the production measurement layer, online evals.