Two daily runs write outputs of the same size and one costs ten times the other, so which per-run numbers separate the causes?

## Why the expensive run looks healthy The signals a running job publishes answer one question: is it working? Throughput in records and bytes, how far behind the input it is, how many units of work failed and were retried, how much a worker wrote to local disk when its working set would not fit in memory. A run that reads forty times more input than its result depends on scores perfectly on every one of them. It is not failing, not lagging, not retrying. It is doing an enormous amount of work correctly, and the only surface on which that is visible is the invoice — which arrives weeks later and, in a shared pool, names the pool. This is the gap that makes a cost review a distinct exercise rather than a by-product of monitoring. ## The four numbers For every run, keep: - **Input bytes read** — the volume actually fetched from shared storage, meaning the storage every machine in the cluster can read, as opposed to a worker's own local disk which vanishes with the worker. - **Capacity held times duration** — worker processes multiplied by the seconds they were held, busy or not. - **Bytes moved between workers** — how much had to be redistributed across the network for steps that need records held elsewhere. - **Output bytes written** — and the number of files, because a destination charges per operation as well as per byte. All four are needed because each is the dominant term for a different failure. Reading too much shows only in the first. A long tail of unfinished pieces shows only in the second, and only as a large product with a small amount of real work behind it. A step that needs records from every other worker — a **wide step** — shows in the third. ## The ratio that actually diagnoses it The single most useful derived figure is **input bytes read divided by the bytes the result genuinely depends on**. A run that produces a daily summary of one day's records but reads three years of them has a ratio in the hundreds. Track its trend per job rather than its absolute level, because the absolute level is meaningless across different jobs and a step change in it is nearly always a real regression. | Symptom | Number that shows it | Likely cause | |---|---|---| | Huge bill, short run | input bytes read | the run read far more than its result needs | | Huge bill, long run, low work done | capacity held times duration | a few unfinished pieces holding the whole cluster | | Huge bill, moderate input | bytes moved between workers | records redistributed across the network at scale | | Bill grew with no code change | input bytes read, over time | the selection the job makes now matches more data | ## Where a tenfold gap usually hides - **A bound that quietly became open-ended.** A selection that once matched one day now matches everything ever stored, and the output is unchanged because the extra rows contribute nothing. - **The same input read more than once in one run.** Each pass is metered again. The mechanism for retaining an intermediate result instead of recomputing it belongs to the memory side of this subject; the bill is simply what makes the second pass visible. - **A tail of a few unfinished pieces.** Four hundred pieces finish in seconds and three run for an hour, and the run is billed for everything it still holds during that hour. - **An input that grew upstream.** The job did not change; the data did. This is the most common answer when two runs of the same code diverge. What you do about the first is largely a storage-layout and selection question owned elsewhere. What this leaf owns is the measurement that tells you it is happening at all. ## What varies between systems Do not assume every engine hands you these four numbers in the same shape. Some publish a run-level total for bytes read; others report only per-step input figures you must sum yourself, and a step that reads the same input twice may be counted once or twice depending on the engine. A continuous job built from repeated small finite runs reports per slice, so a figure for a billing period has to be aggregated across slices. Whatever the shape, capture the numbers as values that outlive the run — a number every worker adds to and the coordinating process sums, so the evidence survives the worker that produced it — rather than reading them from a live screen that will be empty tomorrow. ## How to use it Put one ratio on a recurring review per job — bytes read per unit of useful output — and alarm on a step change rather than on a threshold. A threshold has to be set per job and will be wrong; a doubling week over week is a signal in any job, and it usually arrives with a change you can name.