Two workers must never both charge a customer — what may a lease in a shared volatile tier be responsible for here?
answer
- name the stake before answering
- two holes: deadline passed, claim lost
- the guard lives where the effect lands
- losing the tier should degrade, not break
basics
~10 sIt may be responsible for cost, not for correctness. A lease keeps the common case to one worker, but it cannot promise a single holder, so the guard against a second charge belongs downstream.
solid answer
~50 sA lease is a claim on a key that carries a lifetime, and two workers end up holding one in two established ways: the deadline passes while the first is still working, and a claim acknowledged on a node that is lost before any copy has it disappears when a copy is promoted. In both, neither worker is told. That makes the lease an **optimisation** — almost always one worker does the work, which is worth a great deal when the work is expensive — and not a guarantee. If the effect must not happen twice, the guard goes where the effect becomes durable: a uniqueness rule on the operation's identity at the system of record, or the provider's own duplicate handling. The target is that losing the tier degrades into more duplicate attempts, all rejected downstream, rather than into an outage.
go deeper
The idea worth carrying is that a claim in a fast shared store is a strong hint, not a promise, so anything that must truly happen once is checked where it happens.
Be able to name the two ways two workers hold one claim, and say that the guard against a repeated effect sits at the durable boundary rather than in the claim.
Lead with the stake. Distinguish wasted effort from an effect that must not repeat, and describe the rejection at the system of record that makes the second attempt harmless.
Own the failure test: say what the system does for the ten minutes the tier is unavailable, and design so that answer is a costed degradation rather than an outage or a correctness incident.
## The question behind the question "Is a lease enough?" has no answer until somebody names the stake. As a way of stopping most duplicated work it is genuinely good and cheap. As the only thing between a customer and a second charge it is never enough, and the interview is testing whether you will say the second half out loud. ## Why it cannot be a guarantee The lease's promise is bounded by two well-understood holes, both of which leave two live holders with neither of them informed: - **The deadline passes mid-work.** The claim ends on the store's clock, not on the job's progress. A long pause in the holder's runtime, a lost network path, or simply a run slower than the lifetime allowed produces a second claimant while the first is still executing. - **The claim is lost with the node that accepted it.** Where a write is acknowledged before any copy holds it, the promotion of a copy yields a keyspace in which the claim never existed, and the next conditional create succeeds correctly. Neither hole can be closed by tuning. Renewal shrinks the first and cannot eliminate it; waiting for a copy to hold the claim shrinks the second and cannot eliminate it. A guarantee that exactly one actor acts is a consensus and leader-election problem, with owners and literature of its own, and it is not what an entry in a volatile tier provides. ## Naming the stake The decision is not about the tier at all. It is about what a duplicate costs. | The job's effect | Cost of two concurrent runs | Is the lease enough on its own | |---|---|---| | Rebuilds a derived report, overwriting it | Wasted compute; the result is identical | Yes | | Scans and re-indexes a dataset | Wasted compute, some contention downstream | Yes, usually | | Appends rows with a natural unique identity | Waste, plus a rejection that must be handled | Yes, because the guard already exists | | Sends a notification to a person | Two messages; a trust cost, not a money cost | Depends on the product's tolerance | | Moves money, or calls a provider that does | A second charge, and a customer incident | No, never | The important move is that this table is about the **effect**, not about the store. Teams that argue about how reliable the tier is are usually avoiding the easier conversation about which row they are in. ## Where the guard goes when it must be a guarantee When the bottom row applies, the responsibility moves to the place the effect becomes real: 1. **At the system of record**, as a rule that rejects the second attempt — a uniqueness constraint on the operation's identity, or a conditional state change that only succeeds from the state the first actor left behind. The second worker fails a write instead of duplicating an effect. 2. **At the external party**, using whatever duplicate-suppression the provider offers for a repeated request. This is a contract you read, not a property you assume. 3. **In the shape of the work**, by making the risky step the last one and making everything before it repeatable, so an overlap wastes work rather than producing a second effect. All three share one property: they are evaluated where the effect happens, by the component that owns it. That is what makes them guarantees rather than hints. ## Designing for the tier being absent The last piece is the failure test, and it is the difference between a senior answer and a lead's. Ask what happens when the shared volatile tier is unavailable for ten minutes: - If the workers block until it returns, you have made a convenience into a dependency, and a component chosen for speed now decides your availability. - If the workers proceed with no claim and the downstream guard rejects the duplicates, you have a **degradation with a named cost**: more wasted work, more rejected writes, no incident. - If the workers proceed and nothing downstream rejects anything, the outage is a correctness event, and the design was wrong before the outage started. The second of those is the target. It is also the cleanest argument for keeping the lease even after a downstream guard exists: the guard makes duplicates safe, and the lease makes them rare, which is worth real money when the job is expensive. ## What to say in the room State the stake, name the two holes, put the guard at the durable boundary, keep the lease as a cost control, and say what the system does while the tier is missing. An answer that stops at "we take a distributed claim with a lifetime" is the recipe answer, and the follow-up question is always the same one: what happens when two workers both hold it?
- If a downstream guard rejects duplicates, is the lease still worth keeping?Usually yes, for cost rather than correctness. The guard makes a duplicate safe; the lease makes it rare. When the job is expensive — a long scan, a large rebuild, a paid external call — stopping most duplicate runs before they start is worth far more than the claim costs. Keep it, and stop describing it as a safety mechanism.
- How would you make the second worker's attempt harmless rather than merely unlikely?Give the operation a stable identity and enforce it at the point the effect becomes durable: a uniqueness rule at the system of record, or a state change that can only be applied once from the state the first actor left. The second worker then fails a write rather than producing a second effect, and the failure is visible and countable.
- What should happen while the shared volatile tier is unreachable?Prefer proceeding without claims and letting the downstream guard absorb the duplicates — that is a degradation with a named cost. Blocking every job turns a component chosen for speed into an availability dependency. Whichever you choose, choose it deliberately and write down what the period costs, because the alternative is discovering it during an incident.
saying these in an interview costs you the question
- Calls the lease a guarantee that the charge happens once
- Argues a short enough lifetime makes duplicates impossible
- Puts the guard on the same volatile tier as the claim
- Decides without naming what one duplicate actually costs
- Drops the lease as pointless once a downstream guard exists
- Blocks all work when the claim tier is unreachable