How do you attribute a performance run's slow response tail to one step on the request path when each step's own average looks acceptable?

## Why per-step averages hide the culprit When a path is a sequence of steps and the whole path has a slow tail, the instinct is to rank the steps by their own average duration and blame the top of the list. That instinct fails in two specific and very common ways. **Rare slowness averages away.** A step that responds in 3 ms forty-nine times out of fifty and 800 ms on the fiftieth has a mean near 19 ms — unremarkable next to its neighbours. But it lands on two percent of requests, which is precisely the population that forms the slow end of the path. Its mean says nothing; its own slow end says everything. **Multiplicity averages away too.** A step's *average* is per call, while a request's duration accrues per *invocation*. A step averaging 2 ms invoked sixty times contributes about 120 ms; a step averaging 40 ms invoked once contributes 40. Ranked by average, the second looks three times worse. Ranked by contribution, the first is three times worse. Repeated small calls in a loop are the classic instance and are invisible to any per-call ranking. ## Compare compositions, not aggregates The fix is to stop comparing steps to each other and start comparing the *same* step across different populations of requests. 1. Split the run's completed requests into two slices: the slowest small fraction, and a band around the median. 2. For each slice, sum each step's time and express it as a **share of that slice's total duration**. 3. Look for the step whose share is substantially larger in the slow slice than in the median slice. | Pattern across the two slices | Reading | | --- | --- | | A step's share is much larger in the slow slice | That step is where the slow requests differ | | A step's share is the same in both slices | It is expensive everywhere, not the cause of the tail | | Every step's share is similar, totals differ | Slowness is spread evenly — suspect a shared cause | | Shares sum to well under one in the slow slice | The missing time is between steps, not inside one | The third row usually points at something outside the steps — the requests waited to start, or every step was affected at once. The fourth row is the residual check and it is worth doing every time. ## Weight by how often each step runs Alongside share, record invocations per request for each step, and separate two quantities that get confused: the **per-call cost** and the **calls per request**. A step can dominate a request for either reason, and the remedies are entirely different — make the call cheaper, or stop making so many of them. The composition comparison above naturally captures both, because it works with summed time per request rather than per-call means, but keeping the invocation count beside the share tells you which lever to reach for. ## Correlate within a request, not across the clock A seductive shortcut is to plot each step's slow moments over the run and look for steps whose spikes line up. That grouping finds steps sharing a *cause* — everything was slow at once — but it does not attribute one request's time to one step. Attribution has to be per request: for this specific slow request, where did its time go. Two steps whose slow periods coincide may both be victims of a third thing entirely. ## Mind the residual Sum a slow request's attributed step durations and subtract from its own end-to-end duration. What remains is time the request spent somewhere no step covers: waiting to be admitted before the first step ran, being handed between steps, being serialised on the way out, or inside a stretch nobody measured. A large residual in the slow slice and a small one in the median slice is a strong result in itself — it says the tail is not in any step you are looking at, and it redirects the investigation instead of letting it grind through step rankings that cannot contain the answer. ## Confirm the attribution before acting on it A correlation across two slices is a hypothesis, not a finding. Confirm it by changing the suspect and nothing else, then re-running the identical profile against the same build and data: - Replace the suspect step with a fixed, immediate response and see whether the path's slow end collapses. - Or make its slow behaviour deliberate and worse, and see whether the path's slow end grows in proportion. Either direction is decisive in a way that ranking never is. If the tail is unchanged when the suspect is removed, the attribution was wrong however convincing the shares looked, and the residual is where to look next. This confirmation step is also what separates a diagnosis from a guess when the fix is expensive: it is cheap to run one more comparison and costly to rebuild the wrong step.