How can two workers end up holding the same lease after a failover, when each of them won a conditional create?

## The write that was acknowledged and then was never there The lease recipe rests on one assumption that reads as innocuous: that a successful **conditional create** — a write that succeeds only if the key is absent — means the claim exists. On a single node that is true. Once the tier has copies, the acknowledgement answers a narrower question than the one you asked. It says *the node I talked to had no such key and now holds one*. It does not say how many machines hold it. Put the two facts together and the failure writes itself: 1. Worker A's conditional create is accepted by the node it reached, which answers success. 2. Before that write is on any copy, the node is lost — hardware, host, network, a process killed. 3. A copy is promoted. Its keyspace is the state it had, which does not include A's claim. 4. Worker B attempts the same conditional create against the promoted node. The key is absent, so the create succeeds, correctly. 5. Both workers are now running the job, and both believe they hold the claim. ## Why nobody is told There is no mechanism in this shape that could tell them. The first worker is not waiting on the store; it is running a job. A claim is a passive entry, not a subscription, so there is nothing for the store to deliver even if it knew — and it does not know, because the node that accepted the claim is gone along with the knowledge that it did. The second worker sees an absent key and cannot distinguish *never claimed* from *claimed on a machine that no longer exists*. Renewal does not help either: the first worker's renewals now land on the promoted node, where they find either nothing to extend, or, with a blind renewal, the second worker's claim. ## What genuinely varies between stores This is one of the places where treating one product's behaviour as the model produces a wrong answer: - Some stores in this class acknowledge a write as soon as the receiving node applies it, and copies follow behind. The window is then real on every write. - Others will not acknowledge until at least one copy has the write, either as a deployment-wide posture or as something the caller can request for a particular write. - Some deployments have no copies at all, in which case there is no failover and no promotion — the whole keyspace is simply gone and the claim goes with it, which is a different failure with the same practical result. So the right answer to "how big is this window?" is a question back: what does an acknowledgement mean in this deployment? ## Things that do not help | Proposed fix | Why it does not address this | |---|---| | A longer lifetime on the claim | The claim is absent, not expired; its remaining lifetime is irrelevant to a node that never saw it | | Renewing more often | Renewals cannot reach a node that never held the claim, and blind ones may extend the other worker's | | A holder token on the claim | Protects the release from freeing somebody else's claim; both workers still hold valid, distinct claims | | Writing the claim under two key names | Both writes travel the same path and are lost together | ## What does narrow the window Waiting for at least one copy to hold the claim before treating the create as successful is the honest lever, where the store offers it. It costs latency on every claim and it makes the claim's availability depend on a copy being reachable, which is a trade a team should make deliberately. And it narrows rather than closes: the copy that acknowledged can be the one that is lost, and a promotion decided without agreement among the survivors can still pick a node that never had the write. A guarantee that exactly one worker holds a claim across a failure is a consensus and leader-election problem, and it is not solved by an entry in a volatile tier. ## What to do with that Two practical consequences follow. First, size the blast radius honestly: after a failover, expect a period in which some jobs run twice, and make sure that period is survivable rather than surprising. Second, if the job has an effect that must not happen twice, the claim is not the guard — the guard lives where the effect is made durable, and the claim is there to keep the common case cheap. One related disappearance is worth naming because it looks identical from the worker's side: where the tier is configured to remove entries once it reaches its memory ceiling, a live claim can be removed before its deadline, on a perfectly healthy node. The symptoms are the same — an absent key, a successful create by somebody else, nobody told — which is a good reason to keep claims out of a keyspace that is allowed to shed entries under pressure.