Throughput, Lag and Failures

## What a 'slow' report actually contains 'Slow' is a feeling, not a measurement. Before it becomes a question anyone can answer, three things have to be attached to it: **which part** of the run got slower, **by how much** against a normal run, and **since when**. A *job run* here means one submission of one program to the cluster, from start to its last written output. The numbers below exist to supply exactly those three things, and none of them requires you to know what the program computes. The habit that separates an engineer who can operate a job from one who cannot is reading before changing. Adding machines, raising a memory allowance or increasing the number of pieces the input is cut into are all *changes*, and a change made before a measurement destroys the only comparison you had. ## The five numbers 1. **Processed throughput, as two figures.** Records per second and bytes per second, read separately. They move independently: the record rate can sit flat while bytes per second doubles because the records themselves grew, and the run then dies of volume while the record rate reports nothing unusual. 2. **Duration per step of the job graph.** A *step* is one transformation in the job's graph, applied to every piece of the input. A whole-run duration tells you the run is slower; a per-step duration tells you which single transformation is slower, and that is usually most of the diagnosis. 3. **Distance behind the input** — lag in the sense of unread input: how far the newest record the job has finished with sits behind the newest record its input already holds, in records or in seconds. This figure exists only for a job reading a source that keeps growing. A finite input has an end rather than a growing frontier, so there is nothing to be behind. 4. **Failed and retried units of work.** A *unit of work* is the smallest thing the engine hands to a **worker process** — one process on one machine that runs pieces of the job and owns the memory they use — and the smallest thing the engine will retry by itself. Those retries usually do not surface outside the run, so the count is the only evidence they happened at all. 5. **Memory headroom and bytes spilled.** *Spill* means bytes a worker writes to its own local disk because a working set will not fit in memory. On this reading exercise it is simply a number that has started rising; the memory budget behind it, and what to do about it, belong to the memory subject rather than to this one. ## What each one rules out | Number that moved | Rules in | Rules out | |---|---|---| | bytes/s up, records/s flat | records grew larger; more data per piece | a change in your own program's logic | | one step's duration up, the rest flat | that transformation, its volume, its memory | cluster-wide slowness | | every step's duration up | contention, fewer machines, a shared pool | your program | | distance behind rising | a sustained shortfall between work done and work arriving | a single one-off blip | | retried units up | a sick machine, or a piece of input that keeps failing | a purely arithmetic slowdown | | spilled bytes up | a working set that stopped fitting in memory | a network problem | ## Where the execution model changes the reading This class of engine does not share one runtime, and two of the five figures mean different things depending on the model in front of you. - On a **continuous job built from repeated small finite runs** — the engine slices an endless input into small finite jobs run back to back — durations arrive one slice at a time, and the latency floor is one slice's period. The signal is a slice taking longer than its own period, not the raw duration. - On a **record-at-a-time runtime with key-bound state**, where each record moves through the graph as it arrives and is folded into state held under its key, there are no run boundaries inside the job to time. The equivalent of a per-step duration is the fraction of its time each step spends busy, read continuously. - On the **two-phase disk-to-disk model**, where every phase writes its whole output to shared storage before the next phase reads it, durations come naturally per phase, and one phase may contain several graph steps — so 'the slow step' is a coarser answer there than on the other two. Whatever screen or endpoint a given engine exposes its live and finished-run numbers on, refer to it by what it shows rather than by its name. The names differ between engines; these quantities do not. ## The order is the answer An interviewer asking this is not collecting a list, they are checking whether you have an order and a reason for it. A defensible order runs: rate first (is work still being done at all), then per-step duration (where), then distance behind the input (how bad, and for whom), then failures and spilled bytes (why it may still be getting worse). Answering 'I would add workers' before any of that is precisely the reflex the question exists to catch.