A bad code deploy announces itself. Errors spike, latency climbs, dashboards go red, someone gets paged. A bad model deploy does none of that. The service is healthy, response times are normal, and the answers are subtly worse in ways that show up as a slow erosion of user trust rather than as an incident.
That difference is why AI changes need a rollout process built around detecting silence.
What changed in 2026
- Shadow mode became standard practice. Running a candidate variant against real traffic without serving its output emerged as the low-risk first step in any model change.
- Proxy metrics gained recognition. Behavioral signals — retries, negative feedback, escalation to human support — proved faster and cheaper than judged quality scores for detecting regressions.
- Model versions got pinned and flagged. Treating a model identifier as a feature-flagged configuration value rather than a constant became normal engineering.
- Rollback criteria got written in advance. Defining what would trigger a revert before starting a rollout replaced deciding in the moment.
Rollout stages
| Stage |
Traffic |
What you learn |
| Offline evaluation |
None |
Quality on your test set |
| Shadow mode |
Real inputs, output discarded |
Behavior on production distribution, cost, latency |
| Internal only |
Employees |
Qualitative reactions from people who will tell you |
| Small canary |
A few percent |
Early proxy signals |
| Expanded canary |
A quarter to a half |
Statistical confidence on behavioral metrics |
| Full rollout |
Everyone |
— |
Shadow mode is the underused stage. Running the candidate on real production inputs and discarding its output costs only the inference spend and tells you three things offline evaluation cannot: how it behaves on your actual input distribution rather than your curated test set, what it really costs at your traffic mix, and what its latency profile looks like under production conditions.
What to watch
Quality scores from a judge are useful and slow. The signals that move first are behavioral.
Retry rate is the best single proxy. A user who asks the same thing again, rephrases, or regenerates is telling you the first answer failed, and this shows up within hours rather than days.
Conversation length is a good second signal, and it needs interpretation. Longer conversations can mean engagement or can mean the user is struggling to get an answer. Compare against the baseline variant rather than against an absolute.
Escalation to human support is a direct and unambiguous signal for any assistant that has a handoff path.
Negative feedback, where you collect it, is high-signal and low-volume. Weight it accordingly.
Track cost per successful outcome alongside quality, since a change that improves answers while tripling reasoning tokens may not be worth it — the framing is in AI agent cost per task.
Assign users stickily to a variant. A user bouncing between the old and new model between turns of one conversation gets an incoherent experience, and it also ruins your measurement.
Write the rollback criteria before you start. Specific thresholds on specific metrics, decided while nobody is under pressure, are what prevent the rationalization that sets in when a rollout you have invested in starts looking marginal.
Common mistakes
- Full rollout on the strength of offline evaluation. Your test set is not your traffic.
- Watching only error rate and latency. Both look fine during a quality regression.
- No sticky assignment. Incoherent user experience and contaminated measurement.
- Skipping shadow mode. The cheapest information available and frequently skipped.
- Rollback criteria decided during the rollout. They get negotiated upward.
- Changing model and prompt together. No way to attribute the outcome.
FAQ
How long should a canary run?
Long enough for behavioral signals to accumulate at your traffic volume. For a low-traffic product that may be a week; for high traffic, hours.
Can I canary a prompt change the same way?
Yes, and the same infrastructure serves both. Prompt changes deserve the same caution as model changes.
What if quality is better but cost is higher?
That is a product decision, not a technical one. Quantify both and decide deliberately rather than discovering it on the invoice.
Do I need a feature flag system?
Some mechanism for routing a percentage of traffic by a stable user key. An existing flag system works; so does a simple hash-based router.
Where to go next
For pre-deployment measurement, read prompt A/B testing and eval-driven development for AI. For post-deployment visibility, AI agent observability.