Every health check is green and the process is up, yet consumers say the numbers stopped updating hours ago — what should the alarm have measured?

## Liveness against progress A process being up is the weakest possible claim about a running job. It is also the easiest thing to check, which is why so many jobs are monitored that way and why this scenario is a standing senior interview question. The gap has a precise shape: **liveness** says a process exists and responds; **progress** says useful work is still coming out. Everything that goes wrong in the middle keeps the first true while making the second false: - A unit of work — the smallest thing the engine hands a worker process and the smallest thing it retries by itself — fails and is retried repeatedly on the same poisonous input. The run is alive and permanently stuck. - A step is blocked writing to a destination that has stopped accepting writes. Everything is up; nothing lands. - The source returns no records, either because it genuinely went quiet or because a credential expired and the read is failing silently. The job dutifully processes nothing. - The coordinating process — the single process that turns the program into pieces, hands them out to workers and tracks which finished — is alive, but the workers it needs were reclaimed and never replaced. - The job is producing output, but the output is empty because a filter upstream now matches nothing. Only the last of these is visible in the job's own logs as anything unusual, and none of them is visible in a liveness check. ## Alarm on what the job emits The fix is to measure the thing consumers actually depend on. Three shapes, in rough order of usefulness: 1. **Freshness of the output.** The age of the newest record at the destination, against the wall clock. This is the closest thing to what the complaint was, so it is the closest thing to the right alarm. 2. **Volume against expectation.** The number of results emitted per interval compared with what that interval normally produces. This catches a job that is running and emitting near-empty results, which freshness alone will not. 3. **Distance behind the input crossing a threshold** — lag in the sense of unread input, how far the newest record the job has finished with sits behind the newest record its input holds. This is the earliest of the three on a job reading a growing source, and it exists only on such a job. | Alarm on | Catches | Misses | |---|---|---| | the process being up | the machine or container dying | every form of stuck-but-alive | | the run exiting non-zero | an outright crash | a run that never exits at all | | output freshness | stuck, blocked, starved, crashed | a job emitting fresh but wrong or empty results | | output volume per interval | empty or collapsed results | a slow drift in correctness | | distance behind the input | a growing shortfall, early | a job with no growing input to be behind | ## The two properties that make it work **It must be evaluated outside the job.** An alarm the job raises about itself is not an alarm, it is a log line: a job that has stopped doing anything has also stopped raising alarms. The evaluation belongs to something whose life is independent of the run. **Absence must be the trip condition.** Most checks are written to fire when a bad value arrives. The failure in this scenario produces *no* value, so the condition has to be inverted: fire when no acceptable value has been seen within a window. Practically, that means a check with a deadline attached rather than a comparison against an incoming figure. ## What varies by execution model The words 'the job emits a result' do not mean one thing across this family, and an alarm designed for one runtime misreads another. - On a **continuous job built from repeated small finite runs**, each slice completes and can report success while having processed nothing. A slice-success alarm is nearly worthless here; slice output volume is what you want. - On a **record-at-a-time runtime with key-bound state**, output is continuous rather than periodic, so 'a result per interval' has to be defined by you — usually as a rate over a window — before it can be thresholded. - On the **two-phase disk-to-disk model**, where every phase writes its whole output to shared storage before the next reads it, output arrives in one lump at the end, so freshness at the destination is a coarse signal and the alarm has to be a deadline on the run itself. ## Where this stops being ours A threshold on this job's own signals, or on the result a grouping emits, is an operating concern of the running job. Promises expressed across a *schedule* — this pipeline must have completed by 06:00, this task may be retried three times before the schedule gives up — belong to the system that decides when runs start, which is a different graph with different vocabulary. Keep the two apart in an interview answer; conflating them is the commonest way this question is half-answered.