A cron-style job runs on three nodes and must execute on exactly one. Explain how acquiring a Consul KV key with a session — `PUT /v1/kv/service/job/leader?acquire=<sessionID>` — elects a leader, what happens to that lock when the session is invalidated, and what LockDelay protects against.

## The two objects Leader election in Consul combines a KV key with a *session*. The key is the contested resource; the session is the liveness handle that lets Consul take the key back from a dead holder. ```json PUT /v1/session/create { "Name": "nightly-report", "TTL": "15s", "LockDelay": "15s", "Behavior": "release", "Checks": ["serfHealth"] } ``` The session is bound to the agent that created it and to the checks you list. By default that includes `serfHealth`, the agent's own gossip-derived liveness check, so a node that dies or is partitioned loses its sessions without doing anything. You can add your application's own health check to the list, which is stronger: a process that is running but broken loses the lock too. A `TTL` adds a renewal contract on top — the holder must call `PUT /v1/session/renew/<id>` periodically, and Consul may take up to twice the TTL to invalidate a lapsed one, so size TTLs with that grace in mind. ## Acquiring ``` PUT /v1/kv/service/job/leader?acquire=<sessionID> ``` The body is whatever you want the key to hold — usually the node's identity, so observers can see who leads. The response is `true` or `false`, both under HTTP 200, so parse the body rather than the status. On success the entry's `Session` field names the holder and `LockIndex` increments; `LockIndex` therefore counts how many times the lock has changed hands over the key's lifetime and only ever moves forward. The losing candidates do not spin. They put a blocking query on the key and are woken the instant the `Session` field clears, at which point they race to acquire again. That gives you failover in roughly the time it takes to detect the failure, not a polling interval. ## Losing the lock There are two very different exits. A **clean release** — `PUT /v1/kv/service/job/leader?release=<sessionID>`, or destroying the session — frees the key immediately. The holder chose to step down, so there is no ambiguity about whether it is still working, and re-acquisition can happen at once. An **invalidation** is Consul deciding the holder is gone: `serfHealth` failing, a listed check going critical, or the TTL lapsing. What happens to the key then depends on the session's `Behavior`. With `release` (the default) the lock is freed and the value is kept, which is what you want for leader election. With `delete` the key is removed entirely, which is how you build ephemeral registrations that disappear with their owner. ## Why LockDelay exists This is the part candidates most often miss. When Consul invalidates a session, it has concluded the holder is unreachable — it has not confirmed the holder has stopped. A partitioned leader may still be mid-task, still believing it holds the lock, because it has not yet observed that its own agent lost contact. If a new leader could acquire instantly, both would run. `LockDelay` (15 seconds by default) blocks any acquisition of that key for the delay window following an invalidation. It does not apply to a clean release. The window is a heuristic, not a proof: it gives the old holder time to notice and stop before a new one starts. Setting it to `0` disables it, which is defensible only when the work itself is safe to run twice. ## Advisory, not enforced Consul locks are advisory. Nothing prevents a process that never asked for the lock — or one whose lock was silently invalidated — from writing to the database, sending the emails, or running the report. The lock coordinates only among participants that agree to check it. Three ways to close the gap, in increasing strength: 1. **Make the work idempotent.** Running the job twice produces the same end state. Easiest and most robust. 2. **Re-check leadership before each side effect,** not just once at startup. A long job that verified leadership an hour ago is asserting nothing about now. 3. **Fence.** Carry the key's `LockIndex` — which only increases — into the protected resource, and have that resource reject writes stamped with an index lower than the highest it has seen. This is the only approach that actually stops a stalled old leader that wakes up and resumes, but it requires the downstream system to cooperate. ## What you usually run instead The CLI wraps the whole protocol: `consul lock service/job/leader ./run-report.sh` creates the session, acquires, runs the child while holding the lock, and terminates the child when the lock is lost. `consul lock -n 3 <prefix> <cmd>` builds a semaphore instead, allowing up to that many concurrent holders. Most language clients ship an equivalent helper. Writing the loop by hand is worth doing once to understand renewal and invalidation, and worth avoiding afterwards. ## Sizing the knobs A short TTL fails over fast and risks losing the lock during a stop-the-world pause or a brief network hiccup; a long one leaves the job stalled for longer after a genuine crash. The numbers to reason about together are the TTL plus its 2x grace, then LockDelay on top — that sum is the worst-case gap during which nobody is running the job, and simultaneously the window a wrongly-invalidated leader has in which to notice and stop.