In a job scheduler that hands out time-limited worker leases, how does it detect a worker that died mid-job and reclaim that job?

## Why a lease instead of a 'running' flag In a background-job platform, a worker that dies rarely announces it. A process can be killed by the operating system for using too much memory, the machine can lose power, or a network partition can cut it off while it keeps running. If a job were simply marked `running`, nothing would ever move it again: it would be an **orphan**, stuck forever. A **lease** fixes this by making ownership **time-bound**. The worker owns the job only until `lease_expires_at`. To keep owning it, the worker must keep proving it is alive by **heartbeating**, which means renewing the lease. Silence is the failure signal, and silence covers every failure type at once: crash, hang, kill and partition. ## Claiming and renewing Claiming is one atomic conditional update, so two workers can never both win the same job: ```sql UPDATE jobs SET state = 'running', lease_owner = :worker_id, lease_expires_at = CURRENT_TIMESTAMP + INTERVAL '60' SECOND, attempt = attempt + 1 WHERE job_id = :job_id AND (state = 'queued' OR (state = 'running' AND lease_expires_at < CURRENT_TIMESTAMP)); -- 1 row changed: claim won. 0 rows: someone else holds a live lease. UPDATE jobs SET lease_expires_at = CURRENT_TIMESTAMP + INTERVAL '60' SECOND WHERE job_id = :job_id AND attempt = :my_attempt AND state = 'running'; -- heartbeat; 0 rows changed means the lease was lost ``` Note that `CURRENT_TIMESTAMP` is evaluated by the job store, so every expiry is written and compared on **one clock**. Workers never compare their own wall clock with the stored expiry. ## Reclaiming orphans There are three common ways an orphaned job becomes runnable again: 1. **Reaper scan.** A periodic task finds `running` jobs whose lease has expired, records a `lease_expired` outcome in the run history, and moves them to `queued` or `retry_wait`. 2. **Claim-time takeover.** The claim query itself accepts expired `running` jobs, as in the example above. No separate reaper is needed, although a reaper is still useful for recording outcomes and metrics. 3. **Graceful release.** A worker that is shutting down on purpose clears its lease immediately, so the job does not wait out the full lease. Every reclaim increments `attempt`. That number is what later lets the platform reject writes from the previous holder. ## Choosing lease length and heartbeat interval | Choice | Benefit | Cost | |---|---|---| | Short lease (tens of seconds) | orphans recovered quickly | a long pause or slow network causes a false reclaim and a duplicate run | | Long lease (hours) | almost no false reclaims | a crashed worker's job sits idle for hours | | Heartbeat about every T/3 | a worker can miss a renewal or two and keep the job | more writes to the job store per running job | With illustrative numbers, a 60-second lease and a reaper scanning every 15 seconds, the worst-case detection delay is about 75 seconds: up to 60 for the lease to lapse after the last renewal, plus up to 15 before the next scan. For jobs that run for hours, the right answer is a **short lease with renewal**, not a lease as long as the job. ## Heartbeats in practice - Run the heartbeat on its own timer, separate from the work, so a busy job still renews; but make it stop if the work is truly stuck, or a hung job will hold its lease forever. A progress counter included in the heartbeat lets the platform spot that case. - Treat a heartbeat that changes **zero rows** as a lost lease: stop the work and do not commit. - Batch heartbeats from one worker that runs many jobs into a single write to reduce load. - When a whole pool dies at once, reclaim in batches so survivors are not flooded. ## What a lease does not guarantee - An expired lease means the platform **treats** the job as orphaned. It does not prove the old worker stopped: a paused worker can wake up and try to finish. Guarding commits with the attempt number, known as fencing, handles that. - Because a false reclaim can happen, the same job can execute twice, so job side effects must be idempotent. - A lease detects silence, not wrong answers. A worker that heartbeats while producing garbage needs separate validation.