Records per second is unchanged but each step of the job now takes twice as long — which further measurements separate the causes?

## Two rates that are not the same rate **Processed throughput** is not one number. Records per second counts items; bytes per second counts volume. They move independently, and the gap between them is where most of this diagnosis lives. - Records per second flat, bytes per second up: the records themselves grew. A new field, a nested structure, an upstream producer that started attaching a payload it used not to. - Records per second up, bytes per second flat: more, smaller records. Per-record overhead now dominates, which usually shows up as time rather than volume. - Both flat, duration up: the work per record grew, or the machines got slower. A *step* here is one transformation in the job's graph, applied to every piece of the input, and a *piece* is one slice of that input handled by one worker process — one process on one machine that runs pieces of the job and owns the memory they use. ## The three candidate causes When the record rate holds and per-step duration doubles, the honest shortlist is short: 1. **More data behind the same record count.** Fatter records, or a wider row after a change upstream. The tell is bytes per record. 2. **More data behind the same piece.** The total input grew, or the input is being cut into fewer pieces, so each worker process now holds twice as much. The tell is bytes read per piece and the number of pieces, which are two separate figures and can move in opposite directions. 3. **Slower machines.** Contention from other runs in a shared pool, fewer machines than the run usually gets, or hardware that is degraded. The tell is that *every* step slowed by roughly the same factor, including steps whose input did not change at all. ## Measurements that separate them | Figure to read | Cause it confirms | Cause it eliminates | |---|---|---| | bytes per record, against the last normal run | fatter records | fewer pieces; slower machines | | bytes read per piece, and the piece count | more input per worker | a record-shape change | | the same factor on every step, including trivial ones | slower or fewer machines | anything data-shaped | | duration spread across pieces inside the slow step | a few heavy pieces rather than all of them | a uniform volume increase | | bytes spilled to a worker's local disk | a working set that stopped fitting in memory | a pure arithmetic slowdown | The fourth row is the one candidates skip. A step's duration is usually reported as a total or an average, and an average hides the case where three pieces out of four hundred did all the extra work. Reading the spread costs nothing and changes the answer. ## What varies by execution model The phrase 'per-step duration' does not mean the same thing across this family of engines, so say which model you are on before quoting one. - On a **continuous job built from repeated small finite runs**, where the engine slices an endless input into small finite jobs run back to back, you get a fresh set of per-step durations every slice. Comparing slice against slice is easy, and a slice that outgrows its own period is the real alarm. - On a **record-at-a-time runtime with key-bound state**, where a record moves through the graph as it arrives, there is no repeated boundary to time. What you read instead is the fraction of its time each step spends busy, and the volume figures per step per second. - On the **two-phase disk-to-disk model**, where every phase writes its whole output to shared storage before the next reads it, the natural unit is a phase, which may contain several graph steps. A doubled phase there can hide a single step that got much worse. Whatever figures a given engine publishes, and on whatever screen, the quantities above are the ones to look for; the names attached to them are the engine's business. ## Where the diagnosis hands off This is a reading exercise, and its job is to hand the problem to the right owner rather than to solve it. - A few pieces far heavier than the rest is **uneven work across pieces**, and reading that spread properly is its own subject. - A step whose in-memory queue keeps filling because something after it cannot keep up is **resistance in the graph**, and tracing it back to the step that actually caused it is a separate exercise. - Rising spilled bytes is a **memory** question: how one worker's memory is divided, and what it writes to local disk when a working set will not fit, are owned elsewhere. What this exercise owns is the discipline of never answering 'records per second is fine' as though that settled anything. A job can move twice the data, at the same record rate, with every dashboard reporting a steady figure, right up until a worker runs out of room.