A slowdown reproduces only when requests overlap. How do you design the follow-up performance run that confirms it?

## Why one user cannot see it Some costs exist only in the overlap. An exclusive section over one record, a single cache entry being refilled, a shared counter, an allowance shared across callers, a connection used by more than one request — each costs nothing at all while only one request wants it, and costs real time the moment a second one does. At a concurrency of one the wait is structurally zero: there is never a second holder, so the code path is exercised fully and still looks fast. That is why the usual reproduction attempts fail. Repeating the operation more times does not help, because repetitions are sequential. Running it on a faster machine does not help. Adding recording around the suspect section does not help either, because the section is genuinely quick when uncontended. The only variable that matters is how many requests are inside the shared thing at the same moment, and a single user cannot vary it. ## Vary overlap, hold everything else still The design principle is one manipulated variable. Overlap is the variable; the build, the dataset and its volume, the environment, the request mix and the total work must be identical across the runs being compared, or the comparison decides nothing. The strongest single design separates two things that usually move together: 1. Choose one arrival rate — say two hundred requests per second — and reach it twice. 2. **Run A:** twenty clients, each pausing briefly between requests. Few requests in flight. 3. **Run B:** two hundred clients, each pausing much longer between requests. The same two hundred arrivals per second, roughly ten times the overlap. 4. Compare duration and, if recorded, time spent acquiring the shared thing. If duration is materially worse in Run B at an identical arrival rate, throughput is not the cause — overlap is. This is the design that answers the objection "the system was simply busier", because it was not: it did exactly the same amount of work per second in both runs. ## Two variants that locate what is shared Confirming that overlap matters is half the job; the other half is finding what is being shared. | Run pair | What changes | What a difference proves | | --- | --- | --- | | One arrival rate, few clients vs many | Requests in flight | The cost depends on overlap, not on rate | | Distinct records vs one shared record | What requests collide on | The contention is over that record, not the code path | | Operation alone vs mixed with another | Which operations coexist | The two operations contend over something common | | Concurrency ladder at 1, 2, 4, 8, 16 | Degree of overlap | Whether the cost grows smoothly or has a cliff | The second row is the most decisive and the most often skipped. If the same code path at the same overlap is fast when every request touches a different record and slow when they all touch one, the shared record is the constraint and no amount of profiling the code will show it. The third row catches the case where two operations look individually healthy and only their combination is not. The fourth row distinguishes shapes. A cost that grows gently with overlap suggests a queue in front of something shared. A cost that is flat and then jumps at a particular overlap suggests a fixed allowance being exhausted. A cost that grows faster than overlap suggests the contention itself is adding work — retries, repeated attempts, or a section that gets longer as more callers arrive. ## Make the confirmation readable, not inferential End-to-end duration is a weak signal for this. Before the confirming run, add two recordings around the suspected shared thing: **time spent acquiring it** and **time spent holding it**. Contention shows up unmistakably as acquisition time growing with overlap while holding time stays flat. If instead holding time grows too, the shared section is getting longer under load and the diagnosis changes. Two practical cautions. First, run each configuration more than once and look at the run-to-run spread before believing a difference; a single pair of runs can differ for reasons that have nothing to do with your variable. Second, keep the total work identical: comparing a run that completed a hundred thousand requests against one that completed forty thousand compares two different experiments, however matched their settings looked on paper. What you should end with is a monotone relationship between overlap and cost at a constant arrival rate, and the disappearance of that relationship when the shared thing is removed from the picture. That pair of results is a confirmation. One slow run at high concurrency is not.