Your treatment lifts sign-ups but degrades p95 page latency by 20ms — how do you decide whether to ship?

### Step 1 — is the 20 ms real? Before trading anything away, verify the measurement. - **Look at the interval, not the point estimate.** A reported 20 ms with an interval of [4 ms, 36 ms] is a very different fact from [18 ms, 22 ms]. Latency distributions are skewed, and percentile estimates are noisier than means. - **Check instrumentation symmetry.** If the treatment introduced a new code path, the timer may start or stop at a different point, producing a difference that exists only in the measurement. - **Check for transient causes.** Freshly deployed code can carry cold caches, unwarmed connection pools, or first-run compilation costs that fade. Compare the daily trend across the test window rather than the pooled number. - **Check the invariants.** Skewed traffic split or different bot filtering between arms can move a percentile without any real change in user experience. A guardrail alarm that turns out to be a measurement artefact is common enough that skipping this step is itself a red flag. ### Step 2 — compare against the pre-agreed threshold The decision rule should already exist: a latency budget written down when the experiment was designed, for instance "p95 must not degrade by more than 10 ms". Then the answer is mechanical. - Inside the budget: the guardrail passed. Ship, and record the consumed budget. - Outside the budget: the guardrail is breached. It is not the experiment owner's call to override, which is the entire point of writing the number down beforehand. If no threshold was set, say so honestly — the correct lesson is that the experiment lacked a decision rule — and set one now on the merits, without looking at which side of it the observed value falls. ### Step 3 — decompose the number A pooled p95 is an average over very unequal populations. Two decompositions matter most: - **By device and network.** A 20 ms pooled degradation frequently means a handful of milliseconds on fast desktops and a much larger hit on low-end mobile devices or poor connections. If the harm concentrates on the users least able to absorb it, the pooled number understates the real cost. - **By page or flow.** Latency added to a rarely visited settings page is not the same cost as latency added to the checkout path, even at identical millisecond counts. Also ask about the *shape*: p95 moving while the median is flat means a tail got worse — some requests became much slower. That is a different failure than the whole distribution shifting, and often points at a specific slow path worth fixing. ### Step 4 — quantify the exchange, using your own evidence To weigh a sign-up gain against a latency cost you need a conversion rate between them, and the only trustworthy source is your own product's past latency experiments — deliberate slowdown tests or infrastructure speed-ups measured on the same population. If such evidence exists, the tradeoff becomes arithmetic: the estimated engagement or revenue cost of 20 ms versus the measured sign-up gain. If it does not, be explicit that you are making a judgement call, and avoid importing rules of thumb from other products, where the traffic mix, device profile and baseline latency are all different. ### Step 5 — ask whether the cost is necessary at all Often the most useful move is to reject the framing. Added latency is frequently incidental: a synchronous call that could be deferred, an extra render-blocking asset, a payload that could be paginated. Sending it back with "the feature is good, the implementation costs 20 ms, remove the 20 ms" gets the win without the trade. Alternatives worth naming: ship to a subset while the fix lands, or ship with a monitoring alert on the guardrail so regression is caught in production. ### Step 6 — decide and record The defensible outcomes: 1. **Ship** — degradation inside budget, verified real, no segment concentration. 2. **Ship with a fix** — the gain is real and the latency cost is removable; land the optimisation first or immediately after. 3. **Ship as an explicit tradeoff** — breach accepted by the named owner, written down with the amount of budget consumed, so the next team knows how much room is left. 4. **Do not ship** — the breach is large, concentrated on vulnerable segments, or on a critical path. Whichever it is, record the consumed budget. The systemic risk with latency is a ratchet: every experiment degrades performance slightly, each degradation is individually defensible, and the product gets steadily slower with no single decision to blame. Tracking cumulative consumption against a standing budget is what stops that drift, and it is the part of the answer that marks out someone who has lived through it.