An in-memory store restarts empty and takes traffic again. What does the system of record behind it experience, and how large is that effect?
answer
- the load moves, it does not vanish
- hit ratio sets the size of it
- one divided by the miss rate
- arrives instantly at full arrival rate
- size the system behind for that peak
basics
~20 sUntil entries accumulate again, every read lands on the system of record behind the tier. The multiplier is the inverse of the miss rate: at a 95% hit ratio that system briefly sees twenty times its normal read load.
solid answer
~40 sA tier that comes back empty answers nothing from memory, so for a while every read is served by the system of record behind it. The size of the shock is set by the hit ratio you normally run: at 90% the system behind sees roughly ten times its steady-state read load, at 99% roughly a hundred times. It decays as entries accumulate, but it arrives instantly and at the full arrival rate, which is exactly when that system is least able to shed it. So the capacity behind the tier is sized against the empty-restart peak rather than the steady state, and the restart-time levers are restoring from a copy, pre-loading a chosen key set before taking traffic, and staggering a rolling restart.
go deeper
Recall that a restart can leave the tier holding nothing, and that the reads it was absorbing then land on the system of record behind it.
Turn a hit ratio into a multiplier out loud: the system behind briefly sees roughly one divided by the miss rate times its normal read load, arriving all at once.
Show that capacity behind the tier is sized against the empty-restart peak, and name the restart-time levers - restoring from a copy, pre-loading, and the order and pacing of a rolling restart.
Argue where the cost should sit: permanent headroom behind the tier, engineering effort at restart time, or an accepted degradation window that somebody has agreed to in writing.
## What an empty restart is An in-memory store holds its entries in memory, and memory does not survive the process. When the process stops - a crash, a deploy, a node replacement, an operator restart - everything it held goes with it. What comes back depends on the **posture** the tier was configured with: keep nothing, keep a periodic whole copy, keep a replayed write log, or keep both. This question is about the case where nothing comes back, **an empty restart**, and that case is common for two independent reasons. Some widely deployed stores in this class keep nothing across a restart by design and have no setting that changes that. And plenty of teams running a store that *could* keep something never chose to. The consequence people miss is not inside the tier. The tier is fine: it is empty, it answers quickly, and it fills up again. The consequence lands on **the system of record behind the tier** - the durable engine, service or computation the entries were derived from. ## The arithmetic of the shock In steady state: 1. Reads arrive at the tier at some rate - call it the arrival rate. 2. The tier answers a fraction of them from memory - the **hit ratio**. 3. The system behind therefore sees the arrival rate multiplied by the miss rate, which is one minus the hit ratio. Immediately after an empty restart the tier answers none of them, so the system behind sees the whole arrival rate. The instantaneous multiplier is **one divided by the miss rate**. | normal hit ratio | normally reaches the system behind | multiplier at an empty restart | |---|---|---| | 80% | one read in five | about 5x | | 90% | one read in ten | about 10x | | 95% | one read in twenty | about 20x | | 99% | one read in a hundred | about 100x | The uncomfortable reading of that table is that **the better the tier was doing its job, the worse its absence is**. A tier nobody needed causes no spike when it empties. A tier carrying 99% of reads is a capacity decision wearing the clothes of a latency decision. ## Why the shape is worse than the size - It **arrives at once**. There is no ramp: the first second after the tier opens to traffic already carries the full arrival rate. - It **does not shed itself**. The arrival rate is set by callers, not by the system behind; that system cannot decline work by being slower, it can only queue. - It **decays unevenly**. Popular entries are back in seconds, but the long tail of rarely read keys takes as long as those keys take to be asked for, so a residual elevation outlasts the obvious spike. - It **compounds when callers give up**. Slower answers behind the tier become timeouts in front of it, and timeouts become more arrivals. Designing that protection - timeouts, backoff, circuit breakers - is a resilience-patterns subject rather than a restart decision, but you should expect the effect. - It **hides in averages**. Measured over five minutes the spike looks modest; the system behind lived it in the first twenty seconds. ## What varies between stores - **Whether there is anything to come back from.** Some stores in this class offer a point-in-time copy and a write log; others deliberately keep nothing, and that is a property of the store, not a setting you forgot. - **Whether the process serves while it fills.** Where a copy or a write log is restored, in many designs the process does not accept callers until the load completes, so restoring trades an empty tier for an unavailable one. That **recovery time** is real and grows with how much has to be loaded. - **Whether one node's restart empties the tier at all.** Where the keyspace is split across nodes, restarting one removes roughly its share of the entries. Where each node holds the whole keyspace, restarting one shifts traffic inside the tier rather than behind it. None of that changes the arithmetic above. It changes whose problem the restart is. ## What to do with the number Size the system of record behind the tier against the **empty-restart peak**, not the steady state - or decide explicitly that you accept a degraded window, and say out loud how long it lasts and what it costs. The levers that exist at restart time are **restoring from a copy**, **pre-loading a chosen key set before taking traffic**, and **staggering a rolling restart** by order and pacing. All three move the same quantity: how much of the arrival rate reaches the system behind, and for how long. And this is the reason the tier is never the only copy of anything that matters. An empty restart has to be survivable, not merely unlikely.
- Why does a higher steady-state hit ratio make an empty restart harder rather than easier?Because the hit ratio measures exactly how much work the system of record behind the tier has stopped doing. The multiplier at an empty restart is one divided by the miss rate, so a tier answering 99% of reads is holding back a hundredfold spike, while a tier answering half of them is holding back only a doubling.
- How long does the elevated load behind the tier actually last?Long enough for the entries that are being asked for to be back, which is not one number. The heavily read keys return within seconds and take most of the spike away with them; the long tail returns only as those keys are requested, so a smaller residual elevation can persist for as long as entries normally survive on the tier.
- Does an empty restart matter when the tier holds state that exists nowhere else?It matters more, and differently. There is no system of record behind it to absorb anything, so an empty restart is not a load event at all - it is data loss. That case is decided by the posture the tier runs with, and by whether such state belongs on a volatile tier at all.
A toll plaza where nineteen cars in twenty are waved through by a pass reader and one in twenty queues at a booth. Switch the reader off and every car queues at the booths at once. The traffic did not change - only who has to serve it - and the better the reader had been working, the longer the queue.
saying these in an interview costs you the question
- Says the tier refills gradually, so the system behind never notices.
- Treats a restart as a tier-side problem and never mentions what is behind it.
- Assumes every store in this class can be restored from a copy.
- Estimates the spike from the number of entries rather than the read rate.
- Thinks a high hit ratio makes an empty restart safer rather than sharper.
- Calls the tier a durable place to keep the only copy of something.