A continuous job reports it is four million records behind its input — why does that number alone not say whether it will recover?

## The number in front of you A running job that reads an unbounded input publishes some form of distance behind. The neutral statement of it is **lag in the sense of unread input**: how far the newest record the job has finished with sits behind the newest record its input already holds. It may be published as a count of records behind, or as the age of the oldest record the job has not reached, in seconds behind. Either way it is a **level** — a quantity measured at one instant, like the water sitting in a tank. A level answers `how much work is waiting`. It does not answer the question anybody actually cares about, which is `will this clear`. Two jobs can both read four million records behind at nine in the morning and have opposite futures. One ran all night well above its input rate and is eating through a queue that built up during an overnight peak; it will be clear by ten. The other has been losing a few hundred records a second for a week, and four million is simply where a slow, steady deficit has got to; it will read six million tomorrow. ## Level, trend, and the subtraction underneath The trend is the direction the level moves, and its sign is the sign of one subtraction: **processed throughput, in records per second, minus input rate, in records per second**. Always state the unit and the side, because both quantities get called throughput and the whole subject is the gap between them. | Two readings show | The rates underneath | What it means | |---|---|---| | distance behind falling | processed rate above input rate | an unplanned backlog that is draining; it will clear on its own | | distance behind flat | the two rates are equal | frozen: demand exactly consumes capacity, with no headroom left | | distance behind rising | processed rate below input rate | a real deficit; nothing clears until something changes | The flat case is the one that gets misread. A distance that has not moved all morning is not evidence of health — it says the job is exactly keeping up and has nothing spare, so the next small rise in arrivals, or any slowdown per record, turns it straight into a growing deficit. ## Sampling: how far apart, and against what rhythm The trend is only as good as the interval you sampled it over, and different execution models give the same underlying situation different shapes: - **A job built from repeated small finite runs** — the engine slices an endless input into small finite jobs run back to back — lets unread input climb while a slice is being processed and fall when that slice completes. The line saws. Two readings taken inside one slice show a slope that is not the trend; compare trough with trough, or the envelope across many slices. - **A record-at-a-time job**, where each record moves through the graph as it arrives, produces a smoother line, though on engines where writing a saved snapshot of a running job briefly slows processing, the line ripples at the snapshot interval. - **A two-phase model that writes each phase's whole output to shared storage before the next phase reads it**, launched once per interval, does not move continuously at all: the distance behind is a step function, comparable only at run boundaries. - **Daily rhythm beats all of it.** A job that is behind every afternoon and clear by midnight has no deficit. Sample across a full cycle before calling anything permanent. ## What the trend still does not tell you Three things, and saying so is part of the answer: 1. **Why.** A rising distance says the deficit is real; it does not say whether the input grew, the work per record grew, or capacity was lost. 2. **When it clears.** A falling distance says it will clear, not when. That comes from the gap between the rates, not from the level — dividing the level by processed throughput ignores everything that arrives during the drain. 3. **Freshness.** Zero unread input only means the job has reached the newest record its input holds. If the source itself has gone quiet, the newest record the job has processed can still be hours old. Those are two different senses of lag and they move independently. ## Answering it in an interview Say the level-versus-trend distinction first, then the subtraction, then the sampling caveat. An interviewer asking this is checking one habit: that your first move on a lag alarm is to get a second reading rather than to act on the first.