In a flash sale where any product's cache key can turn hot within seconds, how would you design the policy that promotes keys into a per-process local cache tier?

## What the local tier buys A **per-process local tier** is a small in-memory map inside each application process that sits in front of the shared cache. For a promoted key: - a read first checks the local map and returns immediately on a hit, with no network hop - on expiry, the process reads the shared tier once and stores the value for another TTL - so the shared tier sees about **one read per process per TTL** for that key For example, with 300 processes and a 1-second TTL, a key that clients read 80,000 times per second sends about 300 reads per second to the shared tier (assuming each process refreshes the entry once when it expires). The relief comes from how many processes there are and how long the TTL is, not from how many clients read the key. In a flash sale, the hard part isn't the map. It's the **policy**: which keys go in, when, for how long, and when they leave. ## Deciding which keys may be promoted Not every hot value can tolerate being stale. Classify values before the sale: | Value | Local tier? | Why | |---|---|---| | Product description, images metadata | Yes | Changes rarely, and a second of staleness is harmless | | Displayed price during the sale | Usually | Staleness must be less than the tolerance for price changes | | Sale banner and page shell data | Yes | Read on every page, changes rarely | | Stock count used to accept an order | No | Must be authoritative, otherwise the shop oversells | | Per-user cart or session data | No | Not shared, so there is no hot-key benefit | The **authoritative decision**, such as reserving stock, stays in a transactional path. A cached "only 3 left" label can be approximate. The order check can't be. ## The promotion loop 1. **Detect.** Clients sample requests, estimate per-key rates with a heavy-hitters structure, and report their top keys each window. 2. **Decide.** A collector merges the reports and applies thresholds tied to node capacity. 3. **Publish.** The hot set is pushed to, or pulled by, every process every few seconds. 4. **Serve.** Each process caches promoted keys locally with TTL `T` and a size cap. 5. **Demote.** A key leaves the hot set only after several consecutive windows below a lower threshold. Points that matter in practice: - **Hysteresis.** Promote at rate `H`, demote below `H/2` for 3 windows. Without it, a key near the threshold flaps in and out every window. - **Detection placement.** If counting happens only at the shared tier, promotion hides the key's traffic. Its observed rate collapses, it gets demoted, and it turns hot again. Count at the client, before the local lookup. - **Reaction time.** A detection window of about 1 second plus publishing of about 1 second means about 2 seconds before relief arrives. For keys you *know* will be hot, such as the advertised sale items, **pre-warm** them with a static allowlist. - **Memory cap.** Size the tier explicitly. For example, 1,000 promoted keys at about 20 KB each is about 20 MB per process. Past the cap, keep only the hottest keys. - **Jitter.** Add a small random offset to each process's TTL so the processes don't all refresh at the same instant. ## Bounding staleness The worst-case staleness a reader sees is roughly: - the **local TTL** `T` - plus the **shared tier's staleness** for that key (its own TTL, or how quickly it is invalidated) - plus any **invalidation delay** if you broadcast invalidations to processes Two ways to tighten this bound: - shorten `T`, which raises shared-tier load in proportion (a 250 ms TTL means about 4 reads per second per process) - broadcast an invalidation for promoted keys on write, which shortens the typical staleness, though a lost message still falls back to `T` Write the bound down as a product decision ("the sale page may show a price up to 2 seconds old") rather than leaving it implicit. ## Failure modes - **Detector or collector down.** Fail open: processes keep their current hot set until it expires, then read the shared tier directly. That means less protection, but no outage. - **Hot set too large.** A bad threshold promotes thousands of keys and exhausts process memory. Enforce the size cap and alert on hot-set size. - **Process restarts.** A new process starts with an empty local tier and briefly reads the shared tier directly. A rolling deploy spreads that load. - **Inconsistent views.** Different processes may serve slightly different versions for up to `T`. Users who switch between servers can see a value go backwards, so keep `T` small for values where that matters.