An unreachable host's replicas were replaced; the host returns with its old copies still running and writing - what did the platform actually guarantee?

## The guess at the centre of it Self-healing on an unreachable host rests on an inference the platform cannot verify: *no reports means the workloads are gone*. The deciding half has no contact with the host, which is the entire premise, so it also has no way to confirm the conclusion or to act on it - it cannot stop what it cannot reach. When the timeout expires it does the only thing available: it stops counting those copies and lets the replica loop start new ones elsewhere. If the machine really was dead, that is exactly right. If the network was broken and the machine was fine, the ingester copies on it carried on reading from their sources and writing their output for the entire partition, and now there are more copies than the spec declares. ## Floor, not ceiling This is the precise statement of what a replica count buys you: | The platform does promise | The platform does not promise | |---|---| | To notice when fewer copies exist than declared | That no more than the declared number ever run | | To keep starting copies until the declared number is reached | That a copy it stopped counting has actually stopped | | To stop surplus copies once it can reach them again | Anything at all about a host it cannot reach | The choice is deliberate, and it is a choice for **availability**. A platform that refused to replace until it could prove the originals had stopped would leave workloads short for the entire length of any partition - which is unbounded, since a partition that never heals never proves anything. For interchangeable replicas of a stateless service, running six copies for a few minutes instead of five is harmless, and running three instead of five is an outage. The design picks the harmless failure. ## How the overlap actually ends 1. The host regains contact and its agent re-registers with the control plane. 2. The agent compares what it is currently running against what the control plane says is assigned to it. 3. The copies that are no longer assigned are stopped by the agent, on the host, locally. 4. The workload settles back to the declared count. Note which side performs step 3. The stop is executed by the half that can reach the containers, driven by the state the deciding half holds. That is why nothing can shorten the overlap from the control plane's side: the only actor able to end it is on the far side of the break. A host that never comes back never has its copies stopped by anyone - they end when the machine does. ## Where a real bound has to live If two concurrent instances of this ingester would double-write, corrupt a file, or both act as the single owner of something, the orchestrator is the wrong layer to ask. The bound belongs where the shared resource is: - **A lease with an expiry.** An instance holds a time-bounded claim and must keep renewing it. A partitioned instance cannot renew, so it loses the claim and must stop acting on it before it expires - which makes the safety a property of its own clock, not of anyone reaching it. - **A token the store validates.** Each new holder gets a strictly increasing token; the store rejects writes carrying an older one. A stale instance's writes fail at the point of impact even though it believes it is still in charge. - **Exclusive attachment.** Storage that admits one writer at a time refuses the second attach, so the replacement either waits or fails to start rather than joining the original. - **Work that tolerates duplication.** Making the write idempotent, or keyed so a repeat overwrites itself harmlessly, removes the need for a bound at all - usually the cheapest answer for an ingest path. Each of those is a property of the data path, deliberately outside the scheduler. Note the trade they all share: guaranteeing at most one writer means accepting periods with none. ## What an interview is checking Candidates who have only read about self-healing describe it as a guarantee that the declared number of copies is running. Candidates who have operated through a partition describe it as a guarantee about the lower bound, with an upper bound that is best-effort and briefly violated by design. The second answer is the one that predicts the incident: duplicated readings, a file written by two processes, or a scheduled task that fired twice, all during a window when the dashboard said the workload was healthy at its declared count the whole time.