The database came back at 14:22. By 14:23 it was down again, harder than before. Nothing new broke — the same system that had been running fine an hour earlier could not survive being available.
Everything that had failed during the outage was retrying. Every client that had backed off was backing off on the same schedule, so they all woke up together. Every cache entry that expired while the origin was unreachable was empty, so every request became a cache miss. Load that normally arrives spread across a minute arrived inside two seconds.
This is the thundering herd, and it turns a recoverable incident into a much longer one.
What changed in 2026
- Autoscaling made it worse in a specific way. Systems scale down during an outage because traffic is failing, then face the recovery burst with fewer instances than they had before.
- Serverless amplified retry synchronisation. Thousands of concurrent function invocations retrying on identical schedules produce very sharp spikes.
- Stale-while-revalidate became a default expectation. Serving slightly old data during a refresh moved from a CDN feature to a general application pattern.
- Jitter got built into SDK defaults. More client libraries randomise backoff out of the box, which quietly removed a whole class of incident for teams who never configured it.
Two related failures
They compound and have different fixes.
|
Thundering herd |
Cache stampede |
| Trigger |
Recovery after a failure |
A hot cache key expiring |
| Who piles on |
Retrying clients |
Concurrent requests for one value |
| Arrives at |
The recovering service |
The origin behind the cache |
| Fixed by |
Jitter, backoff, circuit breakers |
Locking, early refresh, stale serving |
| Warning sign |
Synchronised retry schedules |
Identical TTLs, expensive recomputation |
Thundering herd is about clients synchronising. Everyone retried at the same interval, so everyone returns at the same instant.
Cache stampede — also called the dogpile effect — is narrower and often more damaging. One popular key expires. In the milliseconds before anything repopulates it, every concurrent request for that key misses, and every one of them independently starts the expensive computation the cache existed to avoid. A thousand identical database queries fire simultaneously for a value that only needs computing once.
Jitter, and why it does so much
Exponential backoff alone does not solve synchronisation. If every client waits 1s, then 2s, then 4s, they are still perfectly in step — they just take longer to arrive together.
Adding randomness breaks the lockstep. Instead of waiting exactly 4 seconds, wait a random duration up to 4 seconds. Clients spread across the window, load arrives as a ramp rather than a spike, and the recovering service handles it.
This is a small change with a large effect, and it is the first thing to check when a system struggles on recovery. Full jitter — randomising across the whole interval rather than a small band around it — spreads load best.
Two companions matter alongside it. A circuit breaker stops clients from retrying into a service that is clearly down, converting a slow cascading failure into a fast clean one and dramatically reducing the queue that builds up — see the circuit breaker pattern. And a retry budget caps total retries across a client rather than per request, so a widespread failure cannot multiply your own traffic several-fold.
Fixing stampedes
Three approaches, in rough order of how often they are the right choice.
Serve stale while revalidating. When an entry expires, keep serving the old value and trigger a single background refresh. Requests get a slightly outdated answer instead of waiting, and the origin sees one recomputation instead of a thousand. This is usually the best option, and the only question is whether your data tolerates brief staleness — for most cached content it plainly does.
Refresh early, probabilistically. As an entry approaches expiry, give each request a small and rising chance of triggering a refresh. Some single request refreshes it slightly early, and the key never actually expires under load. Elegant, requires no coordination, and works well for hot keys.
Lock the recomputation. The first request to miss acquires a lock and computes; others wait for it or receive stale data. Correct and heaviest, since it needs a distributed lock with careful timeout handling — a lock holder that dies mid-computation must not block everyone indefinitely.
Underneath all three: do not give everything the same TTL. A cache warmed in one batch with a uniform TTL will expire in one batch. Add randomness to expiry times so entries retire gradually. This single habit prevents most stampedes before any other mechanism is needed.
Common mistakes
- Backoff without jitter. Delays the herd instead of dispersing it.
- Uniform TTLs. Guarantees synchronised expiry across everything warmed together.
- No circuit breaker. Clients keep hammering a service that cannot respond, building the queue that overwhelms recovery.
- Retrying non-retryable errors. A 400 will never succeed; retrying it is pure added load.
- Locking without a timeout. A dead lock holder blocks every waiter indefinitely.
- Load testing only steady state. These failures appear at transitions. Test the recovery, not the plateau.
- Scaling down aggressively during an incident. You will need that capacity in ninety seconds.
FAQ
How much jitter is enough?
Full jitter — a random delay anywhere between zero and the current backoff ceiling — spreads load best and is what most guidance recommends. Narrow jitter around a fixed delay helps far less than people assume.
Is serving stale data acceptable?
For most cached content, comfortably. The alternative during a stampede is often serving errors, and a value a few seconds old beats a 503. Where it genuinely is not acceptable, use locking and accept the added complexity.
Does this apply behind a CDN?
CDNs implement request coalescing and stale-while-revalidate precisely because of this, so they absorb much of it. Your origin can still stampede on cache misses the CDN passes through, particularly after a purge — purging everything at once is a self-inflicted stampede.
How does this relate to backpressure?
Backpressure is about refusing work you cannot handle; these patterns are about not generating the work in the first place. Complementary — see backpressure explained.
Where to go next
For the client-side breaker that limits the queue during an outage, read the circuit breaker pattern. For shedding load you cannot absorb, backpressure explained, and for the rate-limiting layer above both, API rate limiting.