Why does a lease key that lets only one worker run a job carry a lifetime instead of living until its holder releases it?

## What a lease is here A **lease** is an entry in a shared volatile tier whose existence means *somebody is working on this*. A worker that wants the job attempts a **conditional create** — a write that succeeds only if the key is absent, in one operation. Whoever wins the create runs the job; everyone else sees the create fail and exits. Nothing about the entry's contents matters yet: its presence is the whole signal. The obvious next step is to delete the key when the job ends, and that is indeed the happy path. The interesting question is what happens when the happy path does not run. ## Release is the path you cannot count on A release is ordinary application code, and ordinary application code does not always execute: 1. The process is killed outright — the host is reclaimed, the container is evicted, a deploy rolls, a supervisor times it out. No cleanup block runs. 2. The process survives but the path to the store does not, and the delete never lands. 3. The process neither dies nor finishes: it hangs on a read with no timeout and sits there for hours. 4. The job throws somewhere the author did not wrap, and the delete is skipped. In each case an entry created with no deadline stays in the keyspace indefinitely. The failure that produces is unusually quiet: nothing crashes. Every later worker attempts the conditional create, fails, logs *somebody else holds it*, and exits successfully. The job simply stops happening. The first real signal is downstream and late — a report that stopped updating, a cleanup that stopped running, a backlog that stops draining — and the repair is a human deleting a key by hand. ## What the lifetime actually promises Attaching a lifetime to the claim converts that permanent, silent outage into a **bounded stall**: - The store, not the worker, measures the deadline. It passes whether or not the job finished, whether or not the worker is healthy, and whether or not anybody is watching. - After a crashed holder, the work resumes at worst one lifetime later, with no human involved. - The lifetime belongs to the entry, not to the job. It does not stretch because the holder is busy, and it does not pause while the holder is blocked. - The claim disappearing is **not** evidence that the job completed. It says only that nobody renewed it. That last point is the one candidates miss. "The claim is gone" and "the job is done" are different statements, and conflating them is how a team ends up treating the tier as a record of work rather than as a hint about who is working. ## Choosing the number The lifetime is a single knob with a cost on both sides. | Lifetime chosen | After a crashed holder | While an unusually slow run is in progress | |---|---|---| | Far longer than the longest run | The job does not run for up to that long, silently | Comfortable: the claim outlives the work | | A little longer than the longest observed run | Stall is short and bounded | One slow run loses its claim mid-job | | Shorter than a normal run | Barely any stall | Routinely two workers on one job | No single value is good in both columns, which is why the honest answer at any scale is a short lifetime that the holder **renews** while it is healthy. Renewal moves the requirement from *longer than the worst run* to *longer than the worst pause between renewals*, which is a smaller and much more stable number. ## What the store has to offer for the recipe to hold The recipe assumes more than every store of this class provides, and it is worth naming which parts: - **A create conditional on absence, in one operation.** Where a store has no such operation, a read followed by a write leaves a gap in which two workers both see an absent key and both write. - **A lifetime attached in the same operation as the create.** Where the lifetime must be a second call, a worker that dies between the two calls leaves exactly the permanent claim the lifetime was meant to prevent, so the two-call version is weaker than it looks. - **A tier that will not remove the entry early.** Where the tier is configured to drop entries once it reaches its memory ceiling, a claim that is still in force can vanish and the holder is told nothing. ## The consequence when the claim is not there When the claim is absent while a worker is still running, the loss is specific: **two workers run the same job at the same time**. If that is duplicated effort — a report built twice, a scan run twice — the cost is waste and the design is fine. If the job has an outside effect that must not happen twice, the claim is an optimisation and the real guard has to live where that effect is made.