After a switch onto a standby cluster, what dominates a stream's recovery time, and why is it not the standby starting up?

## What the recovery-time clock actually measures A **recovery time** is the second of the two numbers you state about a stream: how long until the stream is doing its job again. The only useful definition puts both ends of the clock where the business feels them — it starts when the source cluster stops serving, and it stops when readers on the **standby cluster** are current, not when something accepts a connection. That definition matters because it moves the expensive part of the number out of infrastructure and into the reader side. ## Why the standby starting up is rarely the clock A standby cluster in this arrangement is not a cold thing you boot on the day. It is a running cluster, carrying no writers, kept fed by an ongoing **cross-cluster copier**. It already holds the records, already has its storage attached, already has its own in-cluster copies. Pointing traffic at it is a configuration act, not a provisioning act. (A cluster that is only *built* after the loss is a different and much longer story, and a plan that assumes one should say so.) So the honest clock is made of the stages either side of that: 1. **Detection and decision.** Somebody has to conclude the source cluster is not coming back soon and that switching is better than waiting. On most plans this is a human, and it is measured in minutes that no runbook shortens. 2. **Repointing writers and readers.** Every client has to be told to use the other cluster, and every client has to be allowed to — credentials and access rules on the standby are part of this, and are a common place the clock stalls. 3. **Re-establishing where each reader resumes.** This is the stage with the most variance across platforms, and it is discussed below. 4. **Burning down what accumulated.** While the decision and the switch were happening, writers were failing or buffering and readers were not consuming. Once everything is pointed at the standby, there is a pile to work through before the stream is *current*. This is the stage most often missing from the stated number. ## Where platforms differ on stage 3 | Platform shape | What has to be re-established | What that costs on the clock | |---|---|---| | Readers own a rewindable stored read position | Each reader group's resume point on the target cluster, since position numbers there are the target's own | Real minutes, plus whatever replay or gap the chosen resume point implies | | Records are removed on acknowledgement, consumers compete | Nothing to translate — readers simply begin taking what is present | Reconnection and authorisation dominate instead | | Position is expressed as a timestamp rather than a number | A time to resume from, which travels across clusters more naturally than a number | Cheaper, but only as accurate as the clocks involved | The common thread is that the *records* crossed the hop and the *bookkeeping around them* usually did not. Whatever your platform's answer is, the time it takes belongs in the number. ## Budgeting the number honestly A recovery time that is genuinely defensible is built bottom-up: - Write down each stage above with a measured duration, not an estimate. - Include the decision latency. If the plan says a human declares the switch and that human is on a pager, the pager response time is part of the recovery time. - Size the catch-up stage from arithmetic, not hope: the inbound rate times the outage span gives the pile, and the readers' drain rate gives the minutes. If readers can only consume at roughly the rate producers write, a ten-minute outage takes far longer than ten minutes to clear. - Count the stages you do not own. If producers buffer and then release everything at once, the catch-up burden is larger than the outage; if they drop, it is smaller but you have lost more than the uncopied tail. - State the number per stream. A stream whose readers are a handful of stateless workers recovers on a different clock from one whose reader has to rebuild derived state. ## The classic ways the number is wrong - **Measuring the cluster instead of the stream.** "The standby answers in ninety seconds" is true and irrelevant. - **Omitting the catch-up.** The most common single omission, and the one that turns a fifteen-minute promise into two hours in practice. - **Assuming parallelism you do not have.** On platforms that split a stream into a fixed number of parts, catch-up speed is capped by that count; on platforms where consumers simply compete for work, you can often add readers to drain faster. These are genuinely different recovery curves and the plan should say which one it is on. - **Ignoring the decision.** An automated switch and a human-declared one produce recovery times that differ by more than any technical stage in the list.