How do you sequence a version upgrade across an in-memory tier with a primary and copies that must keep serving?

A version upgrade of a tier that must keep serving is not a deployment, it is a sequence. Everything that makes it hard comes from one fact: each node in this class of store holds its contents in memory, so restarting a node does not restart a process, it empties one. ## The order, and why it is that order 1. **Confirm the preconditions** (below) — this is the only step that is cheap to undo. 2. **Upgrade one copy.** It restarts empty and refills from the node taking writes. 3. **Wait for it to catch up** — replication lag back to its normal range, not merely "connected". 4. **Repeat for the remaining copies**, one at a time. 5. **Hand the write role to an upgraded copy** as a planned, operator-initiated changeover, at a moment when a brief write interruption is cheapest. 6. **Upgrade the former primary**, which comes back as a copy on the new version. 7. **Verify** the whole tier: every node on the intended version, lag normal, connection counts settled, no node holding a stale role. Copies go first because it puts the new version under real traffic in the least consequential position before it is asked to accept writes, and because it costs exactly one write-role changeover instead of several. It also matches the pairing that is more often supported: a copy on a newer version following an older writer. That direction is not universal, and it is a documented property of the store rather than something to assume — confirming it is step 1 for a reason. ## What you confirm before the first restart - **The supported pairing.** Which versions may replicate to which, and how far apart they may be. Some stores support only adjacent versions running together. - **Capacity with a node absent.** On a primary-and-copies shape, memory capacity is unchanged when a copy is down — every node held everything — but read capacity and the failure margin drop. On a partitioned tier, a partition with no copy takes its whole share offline for the length of a restart. - **Whether a rollback exists.** If the older version cannot read what the newer one has written — on disk, or in a changed internal representation — then "go back" means starting the older version empty, which is a different and far more expensive decision. Know that before node one, not after node three. - **The re-sync cost.** One node refilling is a burst of work and bandwidth on the node taking writes; a tier that is already near its limits should do this at a quiet hour. ## What callers see | moment | what callers observe | |---|---| | a copy restarts | connections to that node drop and reconnect; readers of that copy lose it briefly | | a copy is refilling | that node is behind; reading it returns older values until lag settles | | the write role changes hands | a short window where writes fail or wait, and every caller must learn the new writer | | the old primary restarts | one more reconnect burst | Two of these bite. First, reconnects arrive together: every caller that had a connection to the restarted node re-establishes at once, so pools and the server's connection limit both see a spike at the same moment. Second, callers that hold the writer's address rather than resolving a name will keep talking to a node that is no longer taking writes; the changeover is only as clean as the callers' ability to notice it. ## Partitioned tiers and managed instances On a partitioned tier the same rule applies per partition, and you walk partitions one at a time rather than upgrading "all the copies" across the tier at once. A partition whose share has no copy is the case to plan around: either accept that its keys are unavailable for a restart, or move its assignments elsewhere first and bring the node back empty. On a managed instance the sequence may be hidden behind a version control and a maintenance window. What is left to you is real work, not nothing: choosing the timing relative to your own releases, checking that the tier has room, making sure every caller re-resolves the endpoint rather than caching an address, and knowing in advance whether the provider's procedure keeps the data or replaces the instance. ## Where this goes wrong - Restarting several nodes at once, so the writer serves several refills simultaneously and its own service time degrades. - Treating "connected" as "caught up" and moving to the next node while the last one is still behind. - Discovering only at rollback time that the older version cannot read the newer one's data. - Forgetting that the tier's spare capacity has to cover a node being out for the whole procedure, not just for a moment. The habit that makes all of this safe is the same one that makes a capacity move safe: the tier must be able to lose one node comfortably before you deliberately take one away.