Three steps nearest a job's source are each blocked most of the time handing output on; the fourth is not. Which is the bottleneck?

## Why the symptom appears in the wrong place Between each pair of adjacent steps in a job graph there is a **bounded in-job buffer** — a small in-memory queue of records the earlier step has produced and the later one has not yet consumed. When a step slows down, its input buffer fills. Once full, the step feeding it has nowhere to put its next record and stops. That step's own input buffer then fills, and its predecessor stops. This is **backpressure**: a constraint at one point in the graph reproduced as blocking at every point behind it, back to the read from the source. The consequence for diagnosis is the whole content of this subject. **The resistance you notice first is not the resistance that caused anything.** It is a symptom that has travelled. The constrained step is the one that is *not* blocked, because it is busy doing whatever makes it slow. ## The reading rule Order the steps from the source outwards and walk away from the source. Stop at **the first step that is not itself resisting**. Prefer that phrasing over "walk upstream" or "walk downstream", because the two words collide here: upstream in a dataflow means nearer the source, which is where the symptoms are, while people also say "downstream" for the external systems consuming the job's output. Naming the step by its property instead of its direction removes the ambiguity entirely. In the case given, steps one through three are blocked and step four is not, so step four is the constraint. Steps one to three have spare capacity; giving them more of anything changes nothing. ## The patterns and what each means | Pattern across the graph | Where the constraint is | What to ask next | |---|---|---| | Steps 1..k blocked, step k+1 not | Step k+1 | What is step k+1 spending its time on? | | Every step blocked, including the last | The step that writes the job's output, or the destination it writes to | Is the destination rate-limiting, or is the write itself expensive? | | No step blocked, unread input still growing | The read from the source, or the source's own supply rate | How many readers, and can the source serve more? | | Resistance appears and clears on a cycle | An intermittent cause at the first non-blocked step | A recurring heavy piece, a periodic flush, a consumer that throttles in windows? | ## The caveat that separates a middle answer from a rote one The rule is a reading of a pattern, not a law. A step behind the constraint will report blocking only if it can produce faster than the rate it is being held to. A step whose own work is nearly as slow as the constraint spends its time computing rather than waiting, so it shows little or no blocked time even though it sits behind the bottleneck. That is why you look at the *shape* across the whole graph rather than at one number, and why two steps that both look busy are worth measuring against each other before you conclude anything. ## When the figures are not there Not every engine in this family publishes a per-step blocked fraction, and the walk has to be done with whatever is available: - **Buffer occupancy per step**, where it is published: the buffers behind the constraint are full, the one after it is empty. - **Processed throughput per step, in records per second.** In a settled graph every step processes at the same rate, so equal rates tell you nothing by themselves; what distinguishes the constraint is that it is busy at that rate while its predecessors are idle at it. - **Elapsed time per step**, in a model where phases do not overlap. In **the two-phase disk-to-disk model**, each phase writes its whole output to shared storage before the next reads it, so there is no in-flight handover to block and no resistance to propagate at all: the bottleneck is simply the phase with the longest elapsed time. ## Location is not cause Finding the first non-resisting step ends this subject and begins another. That step may be slow because one piece of the input holds far more records than the rest, because its worker process is short of memory, because each record costs a great deal to compute, or because the destination it writes to accepts only so many writes per second. Those have separate owners and separate evidence. The discipline worth demonstrating is refusing to guess among them from the resistance figure alone — the figure named the step, and naming the step is all it can do. ## What this rules out It rules out the reflex of acting on the loudest symptom. The first step, nearest the source, usually looks the worst: it has been blocked the longest and its buffer is fullest. Every instinct says to work on it, and every hour spent there is wasted, because it is idle.