A single unit of a batch job fails identically on every attempt while its neighbours succeed: what does that pattern tell you?

## The two classes, and why the difference is operational - A **transient failure** is caused by the environment the unit happened to run in: the machine disappeared, a network call timed out, a local disk filled, a dependency was briefly unreachable. A fresh attempt elsewhere, or a minute later, has a genuine chance of succeeding. Automatic retry exists for exactly this, and it is why a low attempt cap is a bad idea on a long job. - A **deterministic failure** is caused by something the attempt carries with it: the content of the records in that unit, or a code path only those records take. It reproduces exactly on every attempt and on every machine. Retrying is not merely unlikely to help — it is structurally incapable of helping, because nothing that differs between attempts is what caused the failure. The sharp case of the second class deserves its name: **a record that fails every attempt** — an input whose own content, not the machine, causes the failure. A value the parser never anticipated, a denominator that is zero only in one day's data, a key whose group expands into something the next step cannot hold. ## The evidence that separates them - **Does the failure travel with the machine or with the piece?** Re-runs of the same piece failing on three different hosts, while other pieces on those hosts succeed, is the decisive observation. - **Is the failure point identical?** The same message, at the same place in the code, after the same number of records read, on every attempt. Transient failures are usually ragged; deterministic ones are photocopies. - **Do the unit's neighbours on the same machine survive?** If they do, the machine is not the story. - **Does it still fail hours later?** Time heals a full disk and a flaky dependency. It does not change a value already written into the input. - **Does it fail when that piece alone is re-run in isolation?** This is the cheapest decisive test and is worth naming in the interview, because it separates the two classes without waiting for the job. ## A third case that impersonates the second A piece that is simply too large for the room one worker has will also fail on every attempt and on every machine of the same size — yet the cause is the volume of data, not a value inside it, and the remedies differ (more room per worker, or a smaller piece). How a job handles a piece it cannot fit is its own subject in this tree. Classify by asking whether the failure names **a value** or names **a budget**. ## Why retrying a deterministic failure is worse than wasted 1. It spends the attempt budget that the genuine transient failures elsewhere in the run will need. 2. It costs the unit's full compute on every attempt, and delays everything waiting behind that unit. 3. In a long-running job it is far worse than a wasted unit. Where the recovery unit is a whole connected region rewound to its last recovery point — a durable copy of what the job would otherwise lose, written while it keeps running — the offending record is inside the replayed input, so it is reached again after every rewind. The job does not retry a unit; it restarts, replays and dies at the same place, in a loop whose period is the rewind rather than the unit. ## What actually makes progress | response | what it buys | what it costs | |---|---|---| | Fix the code path the value takes | the record is processed correctly from then on | a deploy, and a rewind or re-run to pick the record up again | | Make the unit tolerate the value: catch, count, continue | the job finishes tonight | records are now being dropped, so the count must be visible and bounded | | Change or repair the input before the job reads it | every downstream consumer benefits | somebody upstream has to own it, and the job waits | Note what is absent from that table: raising the attempt cap. The tolerate-and-count option deserves one caution — a drop rate that climbs run over run means the input has changed shape, not that the job got lucky, so the counter needs a threshold on it. Where rejected records are then sent, and the expectations that would have caught them before the job ever started, are a data-quality subject owned elsewhere. ## What varies between engines - Some engines report which record was in hand when the unit failed; others report only which unit. The first makes this a five-minute diagnosis, the second an afternoon of bisecting the piece. - Where the runtime processes records one at a time as they arrive and holds state between them, there is no per-unit retry to observe: the whole region goes down and comes back, so the same evidence has to be read from the restart history instead of an attempt counter. - In the two-phase disk-to-disk model, where each phase's output is materialised before the next reads it, a deterministic failure in the later phase is reproducible from those durable intermediate files without re-running the earlier phase — a real diagnostic advantage that engines holding intermediates in memory do not have.