As experimentation lead, would you make CUPED the default analysis for every test on the platform?

## The question behind the question This is not really about statistics; the estimator is settled. It is about whether a technique with a metric-dependent payoff and one sharp failure mode should be applied uniformly by a platform, or left to teams. The answer that holds up in an interview is: uniform application is safer than discretionary application, but only if the platform takes on the obligations that discretion was implicitly covering. ## The case for default-on **Free precision where it applies.** For sticky per-user metrics with a good pre-period analogue, the reduction is real and large, and it costs nothing at decision time. **Removing a p-hacking surface.** This is the strongest argument, and it is counterintuitive. A per-team, opt-in adjustment invites the worst possible workflow: run the test, see a null result, try a covariate, report whichever number is significant. The covariate is pre-treatment either way so each individual estimate is unbiased, but *choosing among estimates after seeing them* inflates the false-positive rate. A fixed, platform-computed covariate per metric, locked before the test starts, means there is exactly one number and nothing to select from. **Consistency of the historical record.** If half the experiments are adjusted and half are not, comparisons across time and teams are apples to oranges. ## The obligations that come with it 1. **Pre-registration by construction.** The covariate for each metric is defined in the metric registry, versioned, and cannot be changed mid-experiment. Changing it is a metric change with the same review as any other. 2. **Enforced windows.** The pre-period must close strictly before each user's exposure, not before the analysis date. This is where contamination actually enters, and it should be a property of the pipeline rather than of an analyst's care. 3. **Automated validity checks.** Every experiment reports the arm difference on the covariate itself. A persistent significant gap means either the covariate is contaminated or the assignment is broken, and either should block the readout rather than appear in a footnote. 4. **A/A validation before rollout, and continuously after.** At scale, adjusted effects must centre on zero, the false-positive rate must sit at nominal alpha, and the realised variance ratio should be close to one minus the squared correlation. Replaying historical experiments through the new pipeline and checking that decisions do not flip arbitrarily is a useful complement. 5. **Per-metric honesty about the gain.** Publish the realised correlation and variance ratio per metric rather than a headline number. A metric whose reduction is three percent should say so. ## Where it should not be default Populations with no pre-period history get nothing from it, and quietly shipping a no-op is fine as long as the reporting says so rather than implying a gain. Brand-new metrics have no track record for the correlation, so they start unadjusted until there is historical evidence. Metrics that are already heavily transformed or capped may show much weaker correlation than intuition suggests, which is again a matter for measurement rather than argument. ## The organisational problem nobody mentions The adjusted point estimate is a different number from the naive difference of means that stakeholders see in dashboards. It shifts, sometimes across a decision boundary. If the first time a product manager encounters this is in a launch review, the credibility damage outlasts the precision gain. The rollout plan therefore includes showing both numbers side by side with their intervals for a transition period, a short written explanation of what the adjustment removes, and worked examples from the team's own past experiments. The failure mode to name explicitly is the one where CUPED becomes a knob teams turn after a disappointing readout. Any design that leaves that door open - opt-in adjustment, analyst-selected covariates, re-running with a different pre-period - undoes the statistical benefit and worse, teaches the organisation that the analysis is negotiable. ## The answer in one breath Default-on for a curated metric set with platform-computed, locked covariates and enforced pre-exposure windows; validated by A/A before and monitored after; reported with realised per-metric gains and, initially, alongside the unadjusted number. Never sold as a blanket multiplier on power, and never available as a post-hoc option.