A job outruns its lease's deadline while still working — what does renewing the lease buy, and what can renewal not prevent?

## What renewal changes A claim with a fixed lifetime forces one uncomfortable choice: the lifetime must exceed the slowest run the job will ever have, and that same number is how long the job is stalled after a holder is killed. Renewal breaks the coupling. The holder extends its own claim periodically while it is healthy, so: - the lifetime now only has to cover **the longest pause between two successful renewals**, which is a property of the runtime and the network rather than of the job; - a crashed holder is noticed within roughly one lifetime, so the stall shrinks from minutes to seconds; - an unusually slow run no longer loses its claim just for being slow. Renewal is not free — it is a write per interval per live job — but the volume is trivial next to what it buys. ## Renewal must be conditional too An unconditional extension of the key has the same defect as an unconditional delete: it acts on whatever is under the name. If the deadline already passed and a second worker claimed the key, a blind renewal by the first worker **extends the second worker's claim** — and now the wrong worker's deadline is being pushed forward by a process that no longer holds anything. The renewal must check the holder first: extend only while the stored token still matches. When it does not match, the answer is not to retry harder; it is that the claim is gone. ## The pause renewal cannot cover Here is the sequence that a senior interview is really asking about. The claim carries a thirty-second lifetime and the holder renews every ten seconds: | Time | Worker A | The key in the store | Worker B | |---|---|---|---| | 0:00 | claims, starts work | A's claim, deadline 0:30 | — | | 0:10 | renews | A's claim, deadline 0:40 | — | | 0:20 | host pauses — no renewal leaves the process | A's claim, deadline 0:40 | — | | 0:40 | still paused | absent: the deadline passed | — | | 0:41 | still paused | B's claim, deadline 1:11 | claims, starts the same work | | 1:05 | resumes, believing it holds the claim | B's claim | still working | Nothing here is broken. Each component did exactly what it promised. The store enforced a deadline it was asked to enforce; the second worker won a create against an absent key; the first worker was never contacted, because the store is not in a position to interrupt a caller that is not asking it anything. The result is two workers doing one job, and **neither of them knows it**. The pause can be anything that stops a renewal landing for longer than the remaining lifetime: a long garbage-collection pause, a descheduled container, a host suspended for migration, a network path that drops for twenty seconds, or the store itself being slow to answer. The deadline is measured where the entry lives, not where the worker lives, and beliefs on the worker's side are always an estimate of a decision taken elsewhere. ## Sizing the interval 1. **Fit several renewals inside one lifetime.** An interval of about a third of the lifetime or less means two consecutive failures still leave time to recover. An interval equal to the lifetime means one slow round trip loses the claim. 2. **Set the lifetime from the worst tolerable pause, not from the job.** Ask how long the runtime can stop the process, or the network can be gone, before you would rather somebody else took over. 3. **Treat a renewal attempt like any call to the tier**: give it a timeout shorter than the interval, so a hanging renewal does not silently become no renewal. ## The rule the holder must obey The moment a renewal fails — refused because the token no longer matches, or simply not answered — the worker must stop acting as the holder: - stop starting new outside effects; - checkpoint or abandon the work rather than push it to completion; - do **not** release with a blind delete, because the key may now hold somebody else's claim; - record the event, because it is the measurement that tells you your lifetime is mis-sized. The failure mode teams actually ship is the opposite: a renewal loop that logs a warning and keeps working, on the assumption that the failure was transient. That converts a recoverable overlap into an overlap nobody can see. ## What still varies Stores of this class differ in what they let you express: some can evaluate a condition and extend a lifetime in one server-side step, others cannot, and in those the renewal is a read and a write with a gap between them. Where the condition cannot be expressed at all, renewal is best-effort and the design should lean harder on the downstream guard.