What does a replication-lag figure for an in-memory store actually measure, and how can it look healthy while a copy is useless?

## What the number is computed from Replication lag is the distance between what the primary has accepted and what a copy has received. Two frames are in common use and they are not interchangeable: | Frame | What it says | What it cannot say | |---|---|---| | **Position in the write stream** (a byte or sequence difference) | exactly how much data is outstanding for this copy | how long that will take to close, or how old the copy's data is - both need the current write rate, which is not in the figure | | **Elapsed time** (seconds behind) | how stale the copy's view is right now | anything useful on an idle tier, where nothing outstanding means zero seconds whether the link is alive or dead | A useful habit: quote lag **in the unit it is measured in**, and say which vantage point produced it. A copy's own report and the primary's report about that copy are two different measurements, and they diverge exactly when the connection between them is the problem. ## The three ways this signal lies 1. **It goes blind, not red.** A copy that has disconnected entirely usually stops appearing in the primary's list of copies. Its lag is not reported as enormous - it is not reported at all, and a dashboard that plots the maximum lag across copies will happily show a healthy line computed over the one copy that is still there. This is the single most important property of the signal. 2. **Zero lag on an idle tier means nothing.** Nothing outstanding is not evidence of a working link. A tier that takes writes in bursts can report a perfect number for the entire quiet period between them. 3. **Received is not necessarily applied and servable.** Depending on the store, the figure may track how much of the stream has arrived at the copy rather than how much it has processed and can now answer reads from. Where the two differ, the copy can be simultaneously 'caught up' by the number and behind by what a reader observes. ## Where stores in this class differ - **When the caller's write is acknowledged varies.** Some stores acknowledge as soon as the primary holds the write and stream it onward afterwards; some acknowledge only once a copy also holds it; some let the caller choose per call. The lag figure means a different thing under each: on the first it measures a window of data you would lose, on the second it measures a delay you are already paying on every write. Read the number without knowing which posture you are on and you will misprice a promotion. - **What counts as a copy varies.** Some deployments chain copies behind other copies, in which case the primary's figure describes only the first hop and the lag of the furthest copy is not in it at all. - **Some managed deployments publish only a derived health indicator** rather than the raw distance, and derived indicators usually smooth exactly the short spikes that matter when a promotion is about to happen. ## Alerting on the pair, not on the number - **Alert on the count of connected copies first.** It falling below the intended number is an unambiguous, immediately actionable event, and it is the failure that lag cannot represent. - **Alert on lag second**, as prose arithmetic against a sustained window - for example, a copy more than a few seconds behind, sustained over several minutes, rather than on a single spike, since a brief burst of writes will produce spikes that close by themselves. - **Alert separately on a copy's own report disagreeing with the primary's.** That disagreement is the shape of a link that is up in one direction only. - **Distinguish the dashboard from the pager.** A lag number that rises and falls with write bursts belongs on a dashboard. A copy count below target, or lag that has not closed within the window, belongs on a pager. ## What this number does not license you to conclude The lag figure tells you the **distance** and nothing about the **consequences**. It does not tell you whether an acknowledged write would survive a promotion - that depends on the acknowledgement posture above. It does not tell you whether reads served from the copy are acceptable to the caller, which is a question about the workload rather than the tier. And on a tier whose copies exist for availability rather than for read capacity, a few seconds of lag may be entirely uninteresting right up until the moment a promotion happens, at which point the same number retrospectively describes what was lost. Reading it well means knowing, before the incident, which of those questions your deployment is actually asking.