Forty long-lived replicas of a chat gateway serve two different versions of an edited rate-limit table — how do you detect and bound that split?

## A split fleet is the normal state, not the exception Every mechanism that propagates a value to a running fleet is **per-instance and asynchronous**: - the projection onto each host refreshes on that host's own clock; - each process detects the change on its own watch or receives its own reload signal; - if the change is carried by replacing instances, the replacement walks the fleet a few at a time; - an instance that was unreachable, restarting or newly created at the wrong moment starts from whichever bytes were current for it. So for some window after every edit, part of the fleet enforces the old table and part enforces the new one. The engineering question is never "how do I avoid that?" — you cannot, short of stopping the whole fleet — but "how long is the window, how do I see it, and is the change safe inside it?" A worked bound for the gateway, stating the assumptions: if the change is carried by replacement, a few instances at a time, and each new instance needs roughly twenty seconds to start and be ready, then forty replicas take on the order of several minutes end to end, and **both tables are live for that entire period**. If the change is carried by a re-read, the window is instead the worst-case projection refresh plus the detection delay on the slowest host — usually shorter, but with no natural completion event to observe. ## Why it is invisible by default A replica on the stale table is not failing. It answers every request, returns correct-looking results, passes its health checks and logs nothing unusual. The only symptom is at the edges: - two clients doing the same thing get different answers, depending which replica they landed on; - a limit that was lowered is still not enforced for a fraction of traffic, and the fraction shrinks over minutes; - reverting the edit "fixes" it for some users and not others, which sends the investigation in exactly the wrong direction. That is the real cost of a split fleet: not that it exists, but that nobody can tell it exists. ## Make the version a first-class fact The fix is for each replica to publish **what it is actually serving**, not what was delivered to it: 1. On every successful adoption, compute a short checksum over the raw bytes the process parsed. 2. Store it with the adoption timestamp, and expose it on the instance's status surface. 3. Emit it as a structured field on a log line at adoption, so the history is queryable. 4. Export it as a metric labelled by version, so the fleet-wide count of distinct versions is one series. 5. Alert when that count stays above one for longer than the window you expect for your platform. The checksum must be taken from the bytes the process **adopted**, not from the bytes on disk — otherwise a replica whose reload failed reports the new version while serving the old table, which is worse than reporting nothing. | Signal | What it proves | What it misses | |---|---|---| | Bytes present under the mount | Delivery reached this instance | Whether the process ever read them | | A reload log line | The process attempted a re-read | Whether it adopted or rejected the result | | Adopted-version checksum and time | Which table this replica is serving now | Nothing, if it is emitted only on change and the process later froze | | Distinct-version count across the fleet | Whether a split exists right now | Which replicas, unless it is labelled per instance | Where the workload cannot be changed to report anything, the fallback is behavioural: send probe requests that are shaped to produce a different answer under each version, one per replica, and infer the version from the response. ## Bounding the split by design Detection tells you the split exists; design decides whether it matters. - **Prefer both-tolerant edits.** Lowering a numeric limit, adding a new entry, widening an allowance — both versions serve correctly and the window is merely a period of mixed strictness. - **Split incompatible edits into compatible steps.** To rename a key, first ship readers that accept both names, then change the value set, then remove the old name. Each individual step is safe with a mixed fleet. - **Where the change must be uniform, buy it with a replacement** rather than trusting each process to re-read. A replacement has a definite completion you can observe; a fleet-wide reload has no natural end. - **Do not try to synchronise the moment.** Coordinating forty processes to switch together is a distributed-consensus problem that a configuration edit does not deserve, and its failure mode is worse than the split it prevents. ## What to do while the split is live If a split is discovered during an incident, the first question is which version is correct, and the second is whether waiting closes it. A split created by replacement closes on its own and is observable; a split created by a re-read may be permanent on any replica whose watcher died, and that replica will sit on the old table indefinitely with no error. That is the case where the version report earns its whole cost: without it, the stuck replica is found weeks later by a customer.