Canary testing is the practice of routing a small slice of real production traffic to a new model version while the rest continues to be served by the current one, then comparing results before deciding whether to roll the new version out fully. The name comes from the coal-mine canary: a small, contained exposure that reveals a problem before it reaches everyone.
What changed in 2026
- Canary infrastructure became a standard part of ML deployment tooling, not a custom pipeline every team built from scratch — most serving platforms now support percentage-based traffic splits natively.
- Comparison metrics broadened past accuracy. Latency, cost per inference, and downstream business metrics (conversion, click-through) are now tracked alongside model quality during a canary window.
- Automated promotion rules replaced manual sign-off at teams with mature pipelines — a canary that clears its thresholds for a set window promotes itself.
- Shadow deployment is increasingly used as the step before canary, not instead of it — see our shadow deployment guide for how the two combine.
How a canary test actually runs
- Deploy the new model alongside the current one, both live, both serving real requests.
- Route a defined percentage of traffic to the new version — often starting as low as 1-5% and increasing in steps if it clears each stage.
- Compare metrics between the canary and the control group over a fixed window, long enough to cover normal traffic variation (day-of-week effects, peak vs off-peak).
- Promote, hold, or roll back based on pre-agreed thresholds — not a judgment call made while watching a live chart.
The step that teams skip most often is picking the comparison metrics and thresholds before the canary starts. Deciding what counts as "worse" while you are staring at a slightly-off number in real time leads to both false alarms and missed problems.
Canary vs shadow vs full rollout
| Approach |
Real user impact |
What it tests |
Typical use |
| Shadow deployment |
None — predictions are logged, not served |
Model behavior on live data, without risk |
First check after offline evaluation |
| Canary (small split) |
Limited — a defined slice of real users |
Real-world quality, latency, and business impact |
Gate before full rollout |
| Full rollout |
Complete |
Nothing new — this is the destination, not a test |
After canary clears thresholds |
Shadow testing is lower-risk but cannot measure things that depend on the model actually affecting user behavior, like conversion rate. Canary is the step that catches those.
Picking a traffic split
There is no universal right number. A high-traffic service can canary at 1% and still get a statistically meaningful sample within hours; a low-traffic service may need 20-50% just to gather enough data in a reasonable window. The mistake to avoid is picking a split so small that the canary window has to stretch for days to say anything meaningful — at that point you have delayed the decision rather than made it safer.
When canaries give false signals
Canaries can mislead when the comparison window is too short to average out normal variance, when the traffic routed to the canary is not actually representative (for example, always the same user segment due to sticky routing), or when the new model shifts a metric that recovers over time as it "warms up" caches or personalization state. Build in a minimum window and a sanity check on whether the canary traffic looks like the control traffic before trusting the comparison.
FAQ
What percentage of traffic should a canary start at?
Low enough to limit damage from a bad release, high enough to reach statistical significance in a reasonable time. For most production services, 1-10% is a reasonable starting point, increasing in stages.
How long should a canary run before promotion?
Long enough to cover your traffic's natural variation — usually at least one full day-of-week cycle, sometimes longer for services with strong weekly patterns. Do not promote a canary that has only seen a few hours of one traffic pattern.
Can canary testing replace offline evaluation entirely?
No. Offline evaluation on held-out data is cheaper and catches obvious regressions before any real user is exposed. Canary testing is the next gate, not a replacement for it.
What happens if the canary fails?
It should automatically stop receiving new traffic and revert to the previous version — this is where a solid rollback strategy matters, because the canary failing is exactly the scenario it exists for.
Where to go next