How do you choose the cap for a winsorised revenue metric so the A/B comparison stays fair?

## The threshold is a decision, and it must be made in advance A cap is a parameter of the metric, not of the analysis. Choosing it after seeing the treatment effect is a garden of forking paths: you could try p95, p99, p99.5 and 'a round number', and one of them will flatter the result. That inflates the false-positive rate beyond the nominal level in a way no p-value on the final choice will reveal. So the threshold goes into the metric definition, ideally shared across all experiments on that metric, before any experiment reads out. ## Where the number should come from **Pooled pre-period data is the cleanest source.** Take the spend distribution over the eligible population in a window before the experiment started and read off, say, the 99th percentile. Because that window precedes assignment, the treatment cannot have influenced it, so the transform is guaranteed independent of the outcomes being compared. **Pooled in-experiment data is acceptable but weaker.** A quantile computed on both arms combined is only mildly dependent on the treatment — the dependence vanishes under the null and is usually negligible with large samples — but it makes the transform a function of the data you are analysing, which is harder to defend and harder to reproduce. **Per-arm quantiles are wrong.** If you cap the treatment arm at its own p99 and the control arm at its own p99, you have applied two different transforms. Suppose the treatment genuinely lifts the tail: its p99 is higher, so its cap is looser, and the comparison is no longer like-for-like in either direction you can reason about cleanly. The general rule is that any transform applied to the outcome must be identical across arms and, ideally, fixed before the outcomes exist. ## Which quantile, and at which unit There is no theorem that hands you p99. What you are trading is bias against variance, and the practical way to pick is to look, on historical data, at where the variance reduction saturates. Moving from no cap to p99 usually removes most of the variance; going further to p95 buys much less variance while removing far more of the metric's mass and pushing the estimand further from revenue. Pick the loosest cap that makes the metric decidable at your traffic. Apply the cap at the unit you average over. For a user-randomised experiment analysed on revenue per user, that is each user's total across the experiment window, not each individual transaction — capping transactions and then summing lets a user with many capped purchases still dominate the mean, which defeats the purpose. Also decide the window explicitly: a per-user total over a two-week experiment and a per-user total over six weeks have different distributions, so the same numeric cap means something different. Recompute the threshold when the metric window changes, and refresh it periodically as the spend distribution drifts — with a change log, since a moved cap makes the metric non-comparable to its own history. ## What to report Always publish, next to the result: - the cap value and its provenance (which window, which quantile); - the percentage of users whose value was clipped; - the percentage of total revenue removed by clipping — on a heavy-tailed metric this can be a third even when only 1% of users are touched; - the same clipped fractions per arm, as a sanity check that the two arms are being affected comparably; - the result at neighbouring thresholds. ## Reading the sensitivity ladder Run the analysis at, for example, p99.5, p99 and p98. Three outcomes: 1. **Same sign, similar magnitude, intervals overlapping.** The conclusion is robust; the cap is doing variance control and nothing else. 2. **Magnitude grows steadily as the cap loosens.** The effect has a real tail component; the capped number is a conservative floor, and it is worth saying so explicitly rather than pretending the capped effect is the whole story. 3. **The sign flips.** A handful of users are deciding the answer. Report that you cannot call it, and look at the individual large movers rather than picking the threshold that agrees with you. ## The sentence that scores 'One threshold, pre-registered, derived from data the treatment could not have touched, applied identically to both arms, reported with the share of users and the share of revenue it removes — plus a sensitivity ladder so nobody has to trust the exact quantile.'