A popular cache key expires, and thousands of concurrent requests all miss the cache at the same instant, hammering the database simultaneously. What is this failure mode called, and what techniques prevent it?

## The failure mode **Thundering herd** — also called a **cache stampede** or **dog-piling** — describes what happens when the very thing that makes caching effective (concentrating a large volume of read traffic onto a small number of cached hot keys) turns into a liability at the exact moment that hot key's cache entry disappears. While the entry is cached, thousands of requests per second for the same key are served instantly from memory and the origin sees essentially none of that traffic. The instant the entry expires (or is evicted, or the cache node restarts empty), every one of those requests that would have been a hit is now simultaneously a miss, and because caching logic is typically written per-request ('if not in cache, fetch from database and populate the cache'), every one of those thousands of concurrent requests independently decides to query the database at the same moment. The database, which was previously shielded from nearly all of that traffic, suddenly receives a spike equal to the full uncached request rate for that key — often enough to overload connections, exhaust a connection pool, or dramatically slow down unrelated queries sharing the same database, which in turn slows down the very requests trying to repopulate the cache, sometimes triggering a feedback loop where retries pile on more load faster than the database can drain the backlog. ## Why it is structural, not an edge case The problem exists structurally, not as an edge case, because Zipfian/power-law traffic (a small number of items receiving a disproportionate share of requests — think a viral product page, a celebrity's profile, a trending news article) is exactly the traffic pattern caching is designed to help most, and it's precisely that concentration that makes the expiration moment so dangerous: an unpopular key expiring causes at most one or two near-simultaneous misses, but a viral key expiring can cause thousands. ## The mitigations 1. **Request coalescing, sometimes called 'single-flight'**, is the most direct fix: when multiple concurrent requests miss on the same key, instead of letting each one independently query the database, the caching layer recognizes that a fetch for that key is already in flight and has every other concurrent requester simply wait on (and share) the result of that one in-flight fetch rather than issuing a redundant one. This collapses N simultaneous database queries into exactly one, no matter how large N is. It's typically implemented with a per-key mutex or an in-memory 'promise/future' registry inside the caching layer or application process; libraries like Go's `singleflight` package or Guava's `LoadingCache` in Java implement this pattern directly. 2. **The lock-and-wait pattern** is a related but distinct technique: the first request to miss acquires a short-lived lock (e.g., a Redis SETNX-based lock) and proceeds to query the database and repopulate the cache, while every other concurrent request that finds the lock already held either waits briefly and retries the cache read, or falls back to serving a stale/default value rather than hitting the database itself. This achieves a similar effect to coalescing but is more commonly used across separate processes/servers (not just within one process) since the lock lives in shared cache infrastructure rather than in-process memory. 3. **Jittered TTLs** address a subtly different cause: many systems populate a batch of related cache entries at the same time (e.g., a cache warm-up job, or many keys all set with the same round-number TTL like 'expires in exactly 300 seconds'), which means they also all expire at the same instant, creating a synchronized stampede across many keys simultaneously rather than just one hot key. Adding a small random jitter to each entry's TTL (e.g., 300 seconds plus or minus a random 0-30 seconds) desynchronizes expirations so they spread out over a window instead of firing in lockstep. 4. **Probabilistic early expiration** (sometimes called 'early recomputation' or the XFetch algorithm) takes a proactive approach: rather than waiting for a key to actually expire and letting whichever request happens to arrive first trigger a synchronous refetch, the cache periodically recomputes a probability of early refresh that increases as the entry approaches its real expiration time, and a small fraction of requests near that window trigger an early, asynchronous background refresh — so by the time the entry would have actually expired, it's almost certainly already been refreshed by one of those early, low-probability triggers, and no request ever actually experiences a hard miss on a hot key. 5. **Stale-while-revalidate** takes yet another angle, common in HTTP caching (it's an actual `Cache-Control` directive) and CDN/reverse-proxy configurations: instead of ever exposing a hard miss to the end user, the cache serves the just-expired (stale) value immediately — which is usually 'good enough,' since the data likely hasn't changed drastically in the last few seconds — while kicking off exactly one background refresh to update the cache for subsequent requests. This trades a small, bounded, explicitly-accepted staleness window for guaranteeing the database never sees a stampede at all. ## Layering them in production In production, these are frequently layered: a CDN might use stale-while-revalidate at the edge, an application cache in front of a database might use request coalescing plus jittered TTLs, and a particularly hot key might additionally use probabilistic early expiration so it's essentially never allowed to go fully cold. A well-known real-world example is the concept of 'leases' described in Facebook's Memcache/TAO work — a mechanism functionally similar to lock-and-wait, specifically built to prevent stampedes at Facebook's scale, where a single popular key could otherwise generate an enormous coordinated spike.