Most production AI traffic is not hard. Somewhere between sixty and eighty percent of requests hitting a typical assistant are classification, extraction, short rewrites, or simple question answering — work a small fast model handles indistinguishably from a flagship. Paying flagship prices for all of it is the single most common source of avoidable AI spend.
A model router fixes that by deciding, per request, which tier is sufficient. The idea is obvious. The implementation is where it gets interesting.
What changed in 2026
- Routing moved from a custom build to a product category. Gateways and inference platforms now ship routing as configuration rather than something every team writes from scratch.
- The savings claims got audited. Independent measurements consistently land well below vendor headline numbers, because easy requests are also short requests — they are the majority of calls but a minority of tokens.
- Cascades gained ground on classifiers. Trying the cheap model and escalating on a confidence check turned out to be more robust than predicting difficulty up front, for workloads that tolerate the occasional double call.
- Quality regression monitoring became the hard part. Routing failures are silent. The model answers; it just answers slightly worse, and nothing throws an exception.
Classifier routing vs cascade routing
|
Classifier router |
Cascade router |
| How it decides |
Predicts difficulty before calling |
Calls cheap first, checks, escalates |
| Extra latency on easy requests |
Small classifier overhead |
None |
| Extra latency on hard requests |
None |
Full cheap call wasted |
| Cost on hard requests |
Correct tier immediately |
Pays for both tiers |
| Accuracy of the decision |
Depends on classifier quality |
Grounded in an actual attempt |
| Best when |
Difficulty is predictable from the prompt |
Difficulty only shows in the output |
Classifier routing wins when you can tell hard from easy by looking at the request — long documents, code, multi-step questions. Cascade routing wins when difficulty is invisible up front, which is common in open-ended chat. Plenty of production systems use both: a classifier for the obvious cases and a cascade for the ambiguous middle.
The quality cliff nobody warns you about
The dangerous property of routing is that failures do not announce themselves. When a router sends a request that needed the flagship to a small model, you do not get an error. You get an answer that is subtly worse — a missed nuance, a slightly wrong number, a summary that drops the caveat. Aggregate metrics look fine. User trust erodes anyway.
The only defense is measurement. Sample routed traffic, run the flagship on the same requests offline, and compare. If your small-tier answers diverge materially on more than a few percent of sampled requests, your threshold is too aggressive. This is exactly the discipline described in eval-driven development for AI, applied to a cost decision instead of a feature.
Also decide explicitly what happens on the hard tier when reasoning budgets are involved. Routing and test-time compute interact: the expensive tier is often expensive twice over, once for the larger model and once for the longer thinking. That is fine if it is deliberate and expensive if it is accidental.
Common mistakes
- Routing on prompt length alone. Length correlates weakly with difficulty. A short question can be very hard and a long document can need only extraction.
- Using a large model as the classifier. If the routing decision costs a flagship call, the router has become the expense it was meant to remove.
- No escape hatch for users. Some requests genuinely need the best model and the user knows it. A manual override prevents the worst frustration.
- Ignoring token distribution. Savings track tokens, not request counts. If ninety percent of requests are easy but ten percent carry most of the tokens, your ceiling is much lower than it looks.
- Set and forget. Traffic mix drifts, models get replaced, and thresholds tuned in spring are wrong by autumn.
FAQ
How much can routing realistically save?
For a typical mixed-traffic assistant, forty to seventy percent of model spend. Claims above that usually count requests rather than tokens, or assume a traffic mix skewed further toward trivial calls than most products actually see.
Does routing hurt latency?
Classifier routing adds a small fixed overhead to everything. Cascade routing adds a full wasted call to the requests that escalate. Which is worse depends on your escalation rate.
Should the router be a model or rules?
Start with rules on the obvious signals — request type, document presence, code detection. Add a small model classifier only when rules leave a large ambiguous middle.
What breaks when a provider deprecates a routed model?
Everything, quietly, if you have not planned for it. Keep the tier definitions abstract and see AI model deprecation planning for the migration side.
Where to go next
For the broader cost picture, read AI cost optimization and our AI API cost comparison. If your cheap tier is a small open model, small language models explained covers what that tier can and cannot handle.