The call fails. Somewhere in your request path a model provider returned something other than a completion, and your application has a few hundred milliseconds to decide what happens next. Retry immediately? Wait and retry? Try a different model? Return a cached answer? Tell the user?
Most codebases answer this with a single try/catch that logs the error and returns a generic apology. That is a reasonable place to start and a poor place to stay, because the failures behind it are not one thing.
What changed in 2026
- Refusals became a distinct stop condition rather than an error. A model declining a request returns successfully with a specific stop reason, which means code checking only HTTP status treats it as a normal response and ships an empty answer.
- Server-side fallback appeared as a request parameter. Some providers will re-run a declined request on another model within the same call, which removes a chunk of client-side orchestration.
- Rate limits got more textured. Separate limits for different modes and tiers mean a 429 on one path does not imply the whole provider is unavailable.
- Multi-provider abstraction layers matured. The tradeoff sharpened rather than disappeared: portability against access to provider-specific features, which is a real cost when the features are ones you use.
Classify before you react
The single highest-value change most teams can make is replacing one broad exception handler with a chain that distinguishes cases.
| Failure |
Retryable |
Right response |
| 429 rate limit |
Yes |
Back off, respect retry-after, consider a fallback model |
| 500 / 503 |
Yes |
Retry with backoff; fail over if it persists |
| Timeout / connection |
Yes |
Retry once, then fail over |
| 400 bad request |
No |
Fix the request; log loudly, this is a bug |
| 401 / 403 |
No |
Credential problem; page someone |
| 404 model not found |
No |
Usually a deprecated model — a deploy problem |
| Refusal stop reason |
No |
Different model, different framing, or tell the user |
| Context length exceeded |
No |
Compact, truncate, or split — retrying identical input cannot help |
The two rows people most often get wrong are the last two. Retrying a refusal produces another refusal and bills you twice. Retrying a context-length error with the same input cannot possibly succeed. Both are common in code that catches everything and retries three times by reflex.
Use the typed exception classes your SDK provides rather than string-matching error messages, and order the catch chain from most specific to least. Message text is not a stable API.
Retrying without making it worse
Backoff is not optional at any real volume. Immediate retries against a rate-limited provider extend the outage — every client retrying in lockstep produces a thundering herd that keeps the limit tripped.
Exponential backoff with jitter is the standard answer: wait, then wait longer, and randomise so clients do not synchronise. When the response includes a retry-after header, honour it; the provider knows more than your heuristic. Most official SDKs retry a couple of times by default, which is worth knowing — layering your own retries on top produces a multiplicative total nobody intended, and a request timeout that is really timeout × attempts.
A circuit breaker belongs above the retry logic. If a provider has failed repeatedly in a short window, stop sending traffic for a cooldown rather than retrying every request into a wall. That converts a slow cascading failure into a fast clean one — the circuit breaker pattern covers the mechanics.
Falling back without surprises
Switching to another model is the obvious move and carries a cost that is easy to miss: your prompt was tuned for the original. Formatting compliance, refusal behaviour, and tool-calling reliability all vary between models, and a fallback that produces subtly worse output is harder to detect than one that fails outright.
Run your regression suite against the fallback model, not just the primary. Otherwise the fallback path is the least-tested code in a system that only executes it during incidents — the worst possible combination. LLM regression testing covers making that cheap enough to actually do.
Order fallbacks by similarity, not by price. A smaller model from the same family usually preserves prompt behaviour better than a comparable model from a different provider, even where the benchmark numbers suggest otherwise.
And decide deliberately what "degraded" looks like to a user. A cached or simpler response presented as though it were the full one is a trust problem waiting to surface. Saying "using a faster model right now" costs nothing and is far better received than a silent quality drop that users notice on their own.
Common mistakes
- One catch-all exception handler. Treats a permanent 400 the same as a transient 503.
- Retrying refusals and context-length errors. They cannot succeed and they cost money.
- Stacking your retries on the SDK's. Produces surprising total latency and load.
- No circuit breaker. Every request keeps hammering a provider that is clearly down.
- Untested fallback path. It runs only during incidents, which is when you least want to discover it is broken.
- Silent degradation. Users notice quality drops and draw worse conclusions than the truth.
- Ignoring the refusal stop reason. Code that checks only for HTTP errors will happily return an empty response as a success.
FAQ
Should I use a multi-provider abstraction layer?
It depends on what you use. If your value is in a provider-specific capability, an abstraction that flattens to the common denominator costs you that. If you are doing straightforward completions, portability is cheap insurance. Measure your real failure rate first — many teams build failover for an outage frequency that does not justify it.
How many retries is right?
Two or three, with backoff, for genuinely transient failures. Beyond that you are adding latency to a request that has already taken too long. If a user is waiting, fail fast and let them retry with context about what happened.
What about queuing instead of failing?
Good for asynchronous work, wrong for interactive requests. If nobody is waiting, queue and retry later — that is what batch endpoints are for, per batch vs streaming inference.
How do I test any of this?
Fault injection. Deliberately return 429s and 500s in a staging environment and confirm the behaviour. The failure path deserves the same testing as the success path, and it almost never gets it.
Where to go next
For the circuit-breaking layer above your retry logic, read the circuit breaker pattern. For testing the fallback path properly, LLM regression testing, and for planning the model retirements that turn into 404s, AI model deprecation planning.