You must restart every node of a six-node in-memory tier with no node promoted in its place. How do you choose the order and pacing so the system of record behind it survives?

## First, say what one node's restart actually removes A rolling restart is six restarts in a row, and what each one costs depends on a fact about the tier that has to be stated before anything else: - **The keyspace is split across the nodes.** Restarting one removes roughly its share of the entries. Reads for that share now reach **the system of record behind the tier**, so the spike behind it is roughly the share of the arrival rate that node was serving, multiplied by one over the miss rate. Six restarts means six of those. - **Each node holds the whole keyspace and traffic is spread across them.** Restarting one removes no entries from the tier as a whole. The reads it was serving go to its peers, so what you are managing is concentration inside the tier - remaining capacity and per-node ceiling - not load behind it. The rest of the plan follows from which of those you are in. This question also assumes nobody is promoted in a restarted node's place; arranging for a peer to take over so the restart is invisible is failover, a different subject with its own failure modes. ## The two knobs There are exactly two, and naming them is half the answer: - **Order** - which node goes first. - **Pacing** - how long you wait before the next one. ### Order 1. Rank the nodes by the share of reads they serve, or the share of keyspace they hold. 2. Start with the **smallest share**. The first cycle is where your estimates get tested, and you want them tested where being wrong is cheapest. 3. Measure that cycle end to end: how long the node was unavailable, how long it took before its share of reads stopped reaching the system behind, how high the load behind the tier actually went. 4. Use those measurements to decide whether the plan survives the biggest node, and revise before you get there rather than after. ### Pacing Pace against an **observable behind the tier**, not against a stopwatch and not against process liveness. A node whose process is up but holding nothing is still sending its whole share of reads behind the tier; it is the worst moment of the cycle, not the end of it. The condition for starting the next node is: the previous node is serving, its share of reads is largely being answered from memory again, and the load on the system behind has returned inside its normal headroom. If you cannot see that last quantity, you cannot pace a rolling restart - you can only hope. ## Budget the unavailable window honestly A node is not instantly back. Depending on the tier's **posture**, coming back may include **restoring from a copy** or **replay at start** of a **write log**, and in many designs the process does not accept callers until that finishes. So each cycle is: | phase | what is happening | who feels it | |---|---|---| | stop | the node leaves service | peers, or the system behind | | load | restoring from a copy, or replay at start | nobody is served by this node | | fill | entries accumulate, or a pre-load runs | the system of record behind the tier | | settle | load behind the tier returns to normal | nobody, if you waited | Multiply that by six. The common error is to budget only the stop-and-start and call it a quick cycle, then discover the tier spent the afternoon partially empty. On a store that keeps nothing the load phase vanishes but the fill phase is guaranteed, so pacing matters **more** there, not less. ## The levers that make the cycle cheaper - **Hold the node out of rotation while it fills**, if the routing in front of the tier allows it, and **pre-load a chosen key set** before letting callers reach it. Then the fill phase costs scheduled load instead of arrival-driven load. - **Schedule the whole cycle against the traffic trough.** The multiplier behind the tier is the same at any hour, but the absolute spike scales with the arrival rate. - **Do the arithmetic before the first node**: the share of reads one node serves, times one over the miss rate, against the spare capacity of the system behind. If a single node's share does not fit, no ordering saves you and the answer is pre-loading, a longer window, or not restarting today. - **One node at a time** unless that arithmetic leaves room to spare. Two at once doubles the spike and halves the time you have to notice it. ## What this question is not It is not the miss path. Per-request protection of the system behind - coalescing duplicate work, locking a key while it is rebuilt, short-lived markers for absent entries - is somebody else's design and does not change the order or the pacing. It is also not failover: nothing here is promoted, and every node comes back as itself.