A tier acknowledges writes before a replica holds them: how do you put a defensible number on what a promotion would vaporise?

## Two numbers, not one Asked "how much would we lose", most engineers answer with a duration - "about a second". That is half a number. A defensible answer has two parts: 1. **The magnitude**: how many acknowledged writes no copy held at the worst plausible instant. 2. **The content**: what those writes were, and whether any other system can reproduce them. The first is arithmetic. The second is what turns the arithmetic into a decision, and it is the part that distinguishes someone who has actually run this tier. ## Measuring the lag honestly Propagation lag is exposed differently across stores in this class, and each form has a blind spot: - **Time behind the primary.** Readable and intuitive, but on an idle tier it reads near zero regardless of health, because there is nothing to be behind on. It tells you least exactly when you are calm. - **Outstanding un-propagated work**, as a count of operations or a volume of bytes not yet confirmed. This is closer to what you want, because it is the window itself, but it needs the write rate to become a duration. - **Which side reports it** matters. A number computed on the primary reflects what the primary has sent. A copy that received changes but is slow to apply them can look healthy from the primary's side while holding less than the primary thinks. Where both views are available, take the pessimistic one. Whatever form you get, sample it continuously and keep the distribution. A single spot reading during a calm hour is not evidence about the bad hour. ## From lag to a count of writes The estimate is deliberately crude: ``` writes at risk ~= tail propagation lag (at peak) x write rate (at peak) ``` Use the tail, not the median: 0.4ms of typical lag and 180ms of tail lag are the same deployment, and it is the tail that coincides with the failure you are sizing for - a node under memory pressure, a saturated link, or a burst is exactly the condition that both widens the window and makes a node die. ## What widens it - **A write burst.** Arrivals exceed what propagation can carry, and the backlog grows for as long as the burst lasts. - **A saturated or shared link** between the nodes, including one shared with backup traffic or with another tenant. - **A busy copy.** The lag includes the copy's own conditions - its load, a pause, its own housekeeping. Propagation is not free at the receiving end. - **A long-running single operation on the primary**, which occupies it while writes queue behind it and nothing is sent on. - **A copy that disconnected.** Many stores hold a bounded buffer of recent changes so a copy that drops for a moment can resume from where it stopped. When that buffer overflows, the copy must take a whole fresh copy of the keyspace instead - and for the duration of that transfer it holds nothing recent, so the effective window is the whole transfer. Not every store in this class works this way, and where it does the buffer's size is a capacity decision on the primary, not a free feature. ## The half that decides whether the number matters | What the entries held | What losing the window actually is | |---|---| | Values reconstructable from a system of record | Extra load and a slow period while they are rebuilt | | Advisory counters and rate figures | A small inaccuracy nobody can distinguish from noise | | Deduplication records | Requests processed a second time, with whatever that means downstream | | Claims and leases | Two holders of something meant to have one | | Quota and limit counters | Allowances reset, or consumed twice | The same mechanism and the same count produce an annoyance in the first row and an incident in the last. So the number you defend is not "we would lose 1,200 writes"; it is "we would lose up to 1,200 writes, of which the deduplication records are the ones that matter, and here is what a second delivery of those requests does". ## Defending it A number nobody has tested is a guess. The way to make it defensible is to have caused a promotion deliberately, under representative load, and compared what the new primary held against what callers were told. That exercise also tends to surface the thing the arithmetic misses: whether the write rate at the moment of failure is the steady-state rate at all, since the event that kills a node often arrives together with the burst that overwhelmed it. Finally, keep the frame right. This is a count of writes that were **acknowledged and then ceased to exist** - not a gap to be filled in later from anywhere. That is the property of this tier that makes the number worth computing at all.