Every planned node-by-node cluster change pushes the copies-behind signal past its threshold and pages the on-call, so the team keeps raising the threshold; what should they change instead?

## What raising the threshold actually buys The change is announced, the operator restarts nodes one at a time, the copies-behind signal rises, the pager goes off, and somebody edits the rule. It works, in the sense that the next change is quiet. What it costs is invisible and permanent: the condition that used to notice a replication shortfall at two in the afternoon now needs a much larger one, and the number is never revisited because it is no longer causing pain. Several rounds of this produce a rule whose threshold no real incident can reach. The same logic applies to stretching the sustained window until the change fits inside it. A window long enough to cover a whole rolling change is a window long enough to hide a genuine shortfall for the same duration. ## Two buckets of signal The useful first move is to sort the conditions by whether a planned change is *expected* to move them. | Signal class | Moves during a planned node-by-node change? | Condition treatment | |---|---|---| | Replication shortfall - copies behind | Yes, by design, once per node | Sustained window sized past normal catch-up | | Per-node request latency and saturation | Yes, while load shifts to the remaining nodes | Sustained window, and a threshold set from the reduced-capacity case | | Leadership distribution across nodes | Yes, and it may stay skewed afterwards | Not a page at all; a check after the change completes | | A partition with no node serving it | No - this should not happen even mid-change | Keep it tight, keep it paging | | Writes being refused, or a reader making no progress at all | No - the point of a rolling change is that service continues | Keep it tight, keep it paging | That table is the actual answer to the question. The team has been applying one repair to every row, when the rows need opposite treatment: the top three want a condition that tolerates a bounded, self-healing excursion, and the bottom two want a condition that fires *especially* during a change, because a planned change is exactly when they are most likely to be violated. ## Sizing the window against catch-up For the tolerant bucket, the window is not guesswork: 1. Time a single node's rejoin-and-catch-up on a low-risk change, and record it. 2. Set the window comfortably above the observed worst case for one node - not for the whole change, which is a different and much longer number. 3. Re-measure as the data per node grows. Catch-up time scales with what a node has to fetch, so a window set when the cluster held a tenth of today's data is already wrong. 4. If a single node's catch-up is so long that the required window is unacceptable, that is a capacity finding, not an alerting finding, and it belongs in a different conversation. A shortfall that survives that window during a change is real: it means the node that restarted is not catching up, which is the thing you wanted to know and which the raised threshold would have concealed. ## The bounded suppression, and its limits When the window alone is not enough, suppressing the tolerant bucket for the announced change window is legitimate. Three properties make it safe rather than dangerous: it is scoped to the specific signal and cluster rather than the whole estate, it expires on its own, and it never covers the intolerant bucket. A suppression that silences everything and outlives the change is an outage you have arranged not to hear about. The discipline worth stating out loud is that the temporary and the permanent must not be confused. A threshold change is permanent by default. A suppression is temporary by default. When a team reaches for the permanent tool to solve a temporary problem, the estate slowly loses its alerting without a single decision being recorded. ## What varies across platforms How much a planned change moves these signals depends entirely on the design underneath. On platforms where each partition has a leader and a set of followers that must catch up, restarting a node guarantees a visible shortfall and a visible leadership shift. On platforms where writes are accepted by a majority, the shortfall may never surface as an operator-facing number at all. Where storage is shared rather than replicated per node, there is no catch-up phase to wait out, and the signal that moves is request latency instead. On a rented cluster, the provider may be performing its own changes without telling you, which is its own reason to prefer conditions that tolerate a bounded excursion over conditions that fire on any deviation.