An automatic failover of a volatile tier's primary is often reported as one number; which separate windows does that total outage actually consist of?

## Why one number is the wrong unit Ask how long a failover takes and you will usually be told the time it takes to promote a replica. That is the interval an operator can see in one place, and it is almost never what a caller experienced. Three intervals run back to back, each controlled by a different thing, and only their sum is the outage. ## The detection window Something has to conclude, from silence, that the primary is gone. Whatever the deciding party is — the surviving peers, a set of dedicated observer processes, or a controller outside the data plane — it waits some configured period of unanswered probes before it will act. Stores in this class expose such a timeout precisely so you can set it; what a default is *for* is to be long enough that an ordinary hiccup does not trigger a promotion. This window is bounded below by the longest stall you are prepared to survive without failing over, and the fact that the node is already not answering during it means it is pure outage. It is also the only one of the three that a false positive can start from nothing. ## The promotion window Once the decision is made, a candidate is chosen, reconfigured as primary, and the surviving copies are told to follow it. Some deployments add a check here — the candidate must be current enough, or an operator preference decides between two candidates — and that check costs time. This is the interval most people mean when they quote a failover number, and it is frequently the smallest of the three. ## The rediscovery window The new primary can be accepting writes while every caller is still sending to the dead one. How long that lasts depends entirely on how the caller learns the write address: - a caller holding its own copy of the address refreshes it when it refreshes, which may be on error, on a timer, or never; - a caller resolving a name is bounded by how long its runtime caches the resolution, which is often longer than anyone assumes; - a caller behind an intervening proxy or a stable managed endpoint does not rediscover at all — the address never changed, and the re-pointing happened out of its sight. This is the window operators forget, and the one that most often turns a clean ten-second promotion into a four-minute incident. ## Putting a number on it | Window | Who controls it | What shortens it | What shortening it costs | |---|---|---|---| | Detection | The deciding party's configured timeout | A shorter silence threshold | Failing over on ordinary stalls that would have recovered | | Promotion | The store and the deciding party | Fewer eligibility checks on the candidate | Promoting a copy that is further behind than you would like | | Rediscovery | The caller, or whatever fronts the tier | Re-resolving on failure; a stable address something else re-points | Caller-side work, or a component in the data path to operate | The useful habit in an interview, and in a post-incident review, is to say which of the three your timeout actually controls. A team that halves its detection window and still reports the same outage has usually been paying for rediscovery all along. ## The question to ask of any deployment For each window, ask: what sets it, who can change it, and have we ever measured it? The third question is the one that fails. Detection and promotion can be read out of configuration; rediscovery can only be measured by deliberately failing the primary over and watching how long each caller takes to succeed again. A team that has never run that drill does not know its own failover time, whatever the configuration says. And note what all three have in common on this tier specifically: none of them is a replayable gap in a durable record. The tier is unwritable for the whole sum, and the writes callers attempted during it are simply not there afterwards unless the callers themselves retried. ## One failover, three separately controlled windows. The promotion took two seconds; the outage a caller reports is twelve. The numbers are illustrative — the shape is the point ``` t0 primary stops answering; callers' writes begin failing t0 .. +5s detection window - the deciding party waits out the silence, then concludes the primary is gone t0+5s a surviving replica is chosen as the candidate +5s .. +7s promotion window - candidate reconfigured as primary, remaining copies pointed at it; it now accepts writes +7s .. +12s rediscovery window - callers still holding the old address keep failing, then re-resolve and reconnect t0+12s the last caller is writing to the new primary ```