A nightly job with a very high per-unit attempt cap now finishes four hours late but green: what is the cap hiding?

## What the cap is, and why a generous one is defensible An **attempt cap** is the number of times the engine will re-run a failing unit of work before it abandons the whole run. It exists because machines really do die in the middle of long jobs, and a job of eight hours that fails outright on the first lost worker is worse than useless. So the sensible default is generous, and every operator who has been woken by a job that failed on one flaky host raises it at least once. The cost of that generosity is that the cap does not distinguish failure classes. It counts attempts. A failure that will recur on every attempt — because the cause is the content of the records in that unit — consumes the budget just as happily as a genuine transient one, and it consumes it at full price: the unit's whole compute, every time, plus the delay to everything waiting behind it. ## Why "green" is not the signal The final status of a run answers one question: did every unit eventually produce output? It says nothing about how many times the cluster paid for the ones that did. A run that spent four hours retrying and finished is indistinguishable, at the status level, from a run that had a quiet night. That is the whole trap in this question: **the cap converted a job that is broken into a job that is merely very slow**, and slow does not page anyone. ## The numbers that expose it - **Attempts per unit, not attempts in total.** A total of 200 retries spread over 200 units across 90 machines is ordinary cluster noise. Twelve retries on one unit is a different condition entirely. - **Whether the retries follow the machine or the piece.** Concentration on one piece across several hosts means the failure is carried by the data. - **Time spent in failed attempts against time spent in first attempts.** This is the number that maps directly onto the four missing hours, and onto the bill. - **The trend, run over run.** A retry count that has doubled each week is describing an input that is changing shape, and it will cross the cap eventually — usually on the worst possible night. - **Whether any unit is near the cap.** A unit finishing on attempt eleven of twelve succeeded by luck, not by design. ## A count against a rate | stopping rule | catches | misses | |---|---|---| | a fixed number of attempts per unit | the unit that can never succeed, eventually | a job losing a steady trickle of units to a degraded cluster, since no single unit exhausts its count | | failures over a sliding window of time | a cluster going bad and a tight failure loop, quickly | a rare, slow, genuinely transient failure, which it correctly tolerates | | both together | most real conditions | nothing structural, at the price of two numbers to explain | Engines differ in which of these they offer, and several offer both, so the interview-safe formulation is that a stopping rule wants a *rate* as well as a *count* — the count bounds the work spent on one hopeless unit, and the rate detects a run that is failing steadily without any single unit standing out. ## The same knob in a long-running job Where the recovery unit is a whole connected region rewound to its last recovery point rather than a single unit, the cap governs **involuntary whole-job restarts**, and the arithmetic changes shape. Each attempt now costs a rewind and a replay of everything since that point, not one unit's work. A generous cap here does not produce a slow job; it produces a loop that restarts, replays, reaches the same record and dies, burning far more than it processes. This is the setting where a rate-based rule earns its keep, because a restart loop is visible in attempts-per-minute long before it is visible in a count. ## Two layers, two stopping rules The engine's cap is exhausted entirely inside one run. Only when it is does the run fail — and whatever schedules that run may then decide to run it again later, with rules of its own. Those are different graphs and different owners, and the practical warning is that they multiply: a run that retries a hopeless unit a dozen times, re-run three times by the layer above, has paid for that failure thirty-six times before anyone looks. ## The framing that reads as senior A cap is a **stopping rule, not a diagnosis**. It decides when to stop spending, and it deliberately hides why. Anything that must not be hidden has to be surfaced separately: publish attempt counts as an output of every run, alarm on concentration rather than on totals, and treat a green run that took twice as long as a failure that happened to produce output.