{"post":{"slug":"thundering-herd-and-cache-stampede-2026","title":"Thundering Herd and Cache Stampede: Failures at Recovery","excerpt":"The outage ends and the system immediately falls over again. Everything that was waiting arrives at once, and the recovery is what kills you.","category":"code","date":"August 19, 2026","author":"ByteLedger Team","content":"<span class=\"dropcap\">T</span>he database came back at 14:22. By 14:23 it was down again, harder than before. Nothing new broke — the same system that had been running fine an hour earlier could not survive being available.\n\nEverything that had failed during the outage was retrying. Every client that had backed off was backing off on the same schedule, so they all woke up together. Every cache entry that expired while the origin was unreachable was empty, so every request became a cache miss. Load that normally arrives spread across a minute arrived inside two seconds.\n\nThis is the thundering herd, and it turns a recoverable incident into a much longer one.\n\n## What changed in 2026\n\n- **Autoscaling made it worse in a specific way.** Systems scale down during an outage because traffic is failing, then face the recovery burst with fewer instances than they had before.\n- **Serverless amplified retry synchronisation.** Thousands of concurrent function invocations retrying on identical schedules produce very sharp spikes.\n- **Stale-while-revalidate became a default expectation.** Serving slightly old data during a refresh moved from a CDN feature to a general application pattern.\n- **Jitter got built into SDK defaults.** More client libraries randomise backoff out of the box, which quietly removed a whole class of incident for teams who never configured it.\n\n## Two related failures\n\nThey compound and have different fixes.\n\n| | Thundering herd | Cache stampede |\n| --- | --- | --- |\n| Trigger | Recovery after a failure | A hot cache key expiring |\n| Who piles on | Retrying clients | Concurrent requests for one value |\n| Arrives at | The recovering service | The origin behind the cache |\n| Fixed by | Jitter, backoff, circuit breakers | Locking, early refresh, stale serving |\n| Warning sign | Synchronised retry schedules | Identical TTLs, expensive recomputation |\n\n**Thundering herd** is about clients synchronising. Everyone retried at the same interval, so everyone returns at the same instant.\n\n**Cache stampede** — also called the dogpile effect — is narrower and often more damaging. One popular key expires. In the milliseconds before anything repopulates it, every concurrent request for that key misses, and every one of them independently starts the expensive computation the cache existed to avoid. A thousand identical database queries fire simultaneously for a value that only needs computing once.\n\n## Jitter, and why it does so much\n\nExponential backoff alone does not solve synchronisation. If every client waits 1s, then 2s, then 4s, they are still perfectly in step — they just take longer to arrive together.\n\nAdding randomness breaks the lockstep. Instead of waiting exactly 4 seconds, wait a random duration up to 4 seconds. Clients spread across the window, load arrives as a ramp rather than a spike, and the recovering service handles it.\n\nThis is a small change with a large effect, and it is the first thing to check when a system struggles on recovery. Full jitter — randomising across the whole interval rather than a small band around it — spreads load best.\n\nTwo companions matter alongside it. A **circuit breaker** stops clients from retrying into a service that is clearly down, converting a slow cascading failure into a fast clean one and dramatically reducing the queue that builds up — see [the circuit breaker pattern](/blog/circuit-breaker-pattern-2026). And a **retry budget** caps total retries across a client rather than per request, so a widespread failure cannot multiply your own traffic several-fold.\n\n## Fixing stampedes\n\nThree approaches, in rough order of how often they are the right choice.\n\n**Serve stale while revalidating.** When an entry expires, keep serving the old value and trigger a single background refresh. Requests get a slightly outdated answer instead of waiting, and the origin sees one recomputation instead of a thousand. This is usually the best option, and the only question is whether your data tolerates brief staleness — for most cached content it plainly does.\n\n**Refresh early, probabilistically.** As an entry approaches expiry, give each request a small and rising chance of triggering a refresh. Some single request refreshes it slightly early, and the key never actually expires under load. Elegant, requires no coordination, and works well for hot keys.\n\n**Lock the recomputation.** The first request to miss acquires a lock and computes; others wait for it or receive stale data. Correct and heaviest, since it needs a distributed lock with careful timeout handling — a lock holder that dies mid-computation must not block everyone indefinitely.\n\nUnderneath all three: **do not give everything the same TTL.** A cache warmed in one batch with a uniform TTL will expire in one batch. Add randomness to expiry times so entries retire gradually. This single habit prevents most stampedes before any other mechanism is needed.\n\n## Common mistakes\n\n- **Backoff without jitter.** Delays the herd instead of dispersing it.\n- **Uniform TTLs.** Guarantees synchronised expiry across everything warmed together.\n- **No circuit breaker.** Clients keep hammering a service that cannot respond, building the queue that overwhelms recovery.\n- **Retrying non-retryable errors.** A 400 will never succeed; retrying it is pure added load.\n- **Locking without a timeout.** A dead lock holder blocks every waiter indefinitely.\n- **Load testing only steady state.** These failures appear at transitions. Test the recovery, not the plateau.\n- **Scaling down aggressively during an incident.** You will need that capacity in ninety seconds.\n\n## FAQ\n\n**How much jitter is enough?**\nFull jitter — a random delay anywhere between zero and the current backoff ceiling — spreads load best and is what most guidance recommends. Narrow jitter around a fixed delay helps far less than people assume.\n\n**Is serving stale data acceptable?**\nFor most cached content, comfortably. The alternative during a stampede is often serving errors, and a value a few seconds old beats a 503. Where it genuinely is not acceptable, use locking and accept the added complexity.\n\n**Does this apply behind a CDN?**\nCDNs implement request coalescing and stale-while-revalidate precisely because of this, so they absorb much of it. Your origin can still stampede on cache misses the CDN passes through, particularly after a purge — purging everything at once is a self-inflicted stampede.\n\n**How does this relate to backpressure?**\nBackpressure is about refusing work you cannot handle; these patterns are about not generating the work in the first place. Complementary — see [backpressure explained](/blog/backpressure-explained-2026).\n\n## Where to go next\n\nFor the client-side breaker that limits the queue during an outage, read [the circuit breaker pattern](/blog/circuit-breaker-pattern-2026). For shedding load you cannot absorb, [backpressure explained](/blog/backpressure-explained-2026), and for the rate-limiting layer above both, [API rate limiting](/blog/api-rate-limiting-2026).","hue":320,"metaTitle":"Thundering Herd and Cache Stampede Explained","metaDescription":"Why systems fail on recovery rather than during an outage, how cache stampedes overwhelm an origin, and the jitter and stale-while-revalidate fixes.","keywords":["thundering herd","cache stampede","retry jitter","stale while revalidate","dogpile effect","synchronized retries","cache expiry strategy","resilience patterns 2026"],"tldr":["**The dangerous moment is recovery, not failure** — everything that queued during the outage arrives in one synchronised burst.","**Jitter is the single highest-value fix** — randomising retry delays desynchronises clients for almost no effort.","**A cache stampede is one key expiring under load** — a thousand requests miss simultaneously and all recompute the same value.","**Serve stale while one worker refreshes** — the pattern that eliminates stampedes without adding a distributed lock.","**Skip:** setting identical TTLs across a cache warmed in one batch. They will all expire together and take the origin down."],"readTime":"5 min read","wordCount":1101},"related":[{"slug":"how-to-cache-api-responses-2026","title":"How to cache API responses in 2026: HTTP headers, Redis, and CDN strategies","excerpt":"Caching is the single fastest way to reduce API latency and database load. This is the 2026 guide to HTTP caching, in-process caches, Redis, and CDN layers.","category":"code","date":"March 29, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":315},{"slug":"how-to-cache-with-redis-2026","title":"How to cache with Redis in 2026: patterns, TTLs, and invalidation","excerpt":"Redis caching in 2026 is faster to set up than ever, but the same mistakes persist: missing TTLs, stale data, and cache stampedes. This guide fixes all three.","category":"code","date":"March 11, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":307},{"slug":"llm-fallback-strategies-2026","title":"LLM Fallback Strategies: Staying Up When a Model Is Not","excerpt":"Your provider rate-limits you, or a model refuses, or an endpoint goes down. What your app does in the next 200 milliseconds decides whether users notice.","category":"code","date":"August 18, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":318}],"sameCategory":[{"slug":"bazel-build-explained-2026","title":"Bazel Explained: What Hermetic Builds Actually Cost","excerpt":"Bazel delivers reproducible, precisely cached builds and demands that you describe everything. Here is what you get, what it costs, and who should say no.","category":"code","date":"August 19, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":145},{"slug":"graphql-federation-2026","title":"GraphQL Federation: One Graph, Many Teams","excerpt":"Federation composes independently-owned subgraphs into a single API. Here is what it solves, the operational cost it adds, and when a simpler gateway is enough.","category":"code","date":"August 19, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":300},{"slug":"hermetic-builds-explained-2026","title":"Hermetic Builds Explained: Same Inputs, Same Output","excerpt":"A hermetic build depends only on declared inputs. Here is what that property gives you, the common leaks that break it, and how to move toward it incrementally.","category":"code","date":"August 19, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":130},{"slug":"monorepo-tooling-compared-2026","title":"Monorepo Tooling Compared: What You Actually Need","excerpt":"Monorepo tools solve task orchestration and caching, not repository layout. Here is what each tier provides and how to tell when you have outgrown the simple option.","category":"code","date":"August 19, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":150}],"newer":{"slug":"thread-border-router-explained-2026","title":"Thread Border Routers: The Smart Home Part Nobody Explains","excerpt":"You bought a Matter device and it will not connect. The missing piece is usually a border router, a thing you may already own and have never heard of.","category":"tech","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":230},"older":{"slug":"zfs-explained-2026","title":"ZFS Explained: The Filesystem That Checks Its Own Work","excerpt":"ZFS detects and repairs silent data corruption that ordinary filesystems never notice. Here is what it provides and the memory question people argue about.","category":"tech","date":"August 19, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":225}}