Which signals on a running job move first when it degrades, and which only confirm the damage afterwards?

## Why the split matters Every number on a running job answers a question about time as well as about health: *how long after the underlying problem starts does this figure move?* Sorting the small set of available figures on that axis is what lets you say 'I would look at this one first' and defend it — which is exactly what an interviewer is after. A *leading* signal changes while the job is still meeting its obligations. A *lagging* signal is an accumulation: it integrates a shortfall over time, so by the time it is alarming, the shortfall has been running for a while and the damage is already done. ## The signals, sorted | Signal | Leading or lagging | Why it sits there | |---|---|---| | bytes spilled to a worker's local disk | leading | appears the moment a working set stops fitting, long before output slows | | memory headroom on a worker process | leading | falls continuously toward the point where a unit of work fails | | retried units of work | leading | a failing machine or a bad piece of input shows here before the run's duration moves | | reported resistance on a step | leading | a step already spends time waiting to pass output on before any consumer notices | | processed throughput | mixed | responds immediately to a change, but a healthy figure proves nothing about whether it is enough | | distance behind the input | lagging | an accumulation of the gap between work done and work arriving | | wall-clock duration of the run | lagging | known in full only at the end | | output freshness as the consumer sees it | lagging | the last thing to move and the first thing anyone complains about | Definitions used above, since the names differ between engines: a **worker process** is one process on one machine that runs pieces of the job and owns the memory they use; a **unit of work** is the smallest thing the engine hands a worker and the smallest thing it retries by itself; **spill** is bytes a worker writes to its local disk when a working set will not fit in memory; and *distance behind the input* is lag in the sense of unread input — how far the newest record the job has finished with sits behind the newest record its input already holds. ## The trap inside the lagging column Distance behind the input is lagging evidence *of the job's own health*, and people over-read it in both directions. - It rises when the job slows, slowly, because it accumulates. - It also rises immediately when the **input rate** jumps, with the job performing exactly as it always has. That is a change in the world, not a degradation. - It can be flat and large: a job holding a steady deficit is keeping up with arrivals while remaining permanently stale. - It can be zero while the newest record the job has processed is hours old, because the source went quiet. That is a freshness question about event time and belongs to a different subject; do not read it as the job being healthy or unhealthy. So distance behind the input is a good alarm for the consumer's promise and a poor first diagnostic for the job. ## What varies across execution models Which leading signals you actually get depends on the runtime, and an answer that assumes one model is wrong for the next. 1. On a **continuous job built from repeated small finite runs** — an endless input sliced into small finite jobs run back to back — the earliest signal is usually a slice taking longer than its own period. It is leading because the slices have not yet piled up. 2. On a **record-at-a-time runtime with key-bound state**, the earliest signal is the per-step busy fraction and the fill level of the in-memory queue between two steps, because records are pushed onward as they are produced. 3. On the **two-phase disk-to-disk model**, where each phase writes its whole output to shared storage before the next reads it, there is very little in flight to watch: the useful leading signals are the retried-unit count and spilled bytes, and the phase boundary is where you learn the rest. ## How to use the split - **Alarm on the lagging ones**, because they express the promise you made to a consumer, and a promise is what an alarm should protect. - **Investigate with the leading ones**, because they move early enough to act on, and because they point at a mechanism rather than at a symptom. - **Never alarm on a leading signal alone** without a duration qualifier. Spill happens by design in a healthy job; a retried unit is normal; a step waiting briefly is normal. It is spill that keeps growing, retries that keep repeating and waiting that becomes constant that mean something. What this leaf owns is the sorting. Why a particular step is the one under resistance, and whether a growing deficit will ever drain, are the two neighbouring exercises this one hands off to.