When would you use pre-experiment stratification on platform or country instead of a CUPED covariate?

## Two routes to the same goal Variance reduction in an experiment always works the same way: identify variation in the metric that something already known explains, and stop paying for it in the error bars. The two mechanisms differ in what that known thing is and at which stage it acts. **Stratification** partitions users into discrete cells before assignment - platform, country, maybe a coarse tenure bucket - and randomises within each cell. The arms then hold the same mix of cells by construction. **CUPED** leaves assignment alone and subtracts, at analysis time, a scaled pre-period covariate from each user's outcome. ## The ceiling on stratification Decompose total metric variance into a between-cell part and a within-cell part. Stratification removes only the between-cell part - the piece attributable to the fact that, say, one platform's users are systematically heavier spenders than another's. Whatever spread remains among users inside the same cell is untouched. For most per-user product metrics that decomposition is unkind. Revenue per user, sessions per user and time spent are heavy-tailed and dominated by individual propensity. Two users on the same platform in the same country can differ by an order of magnitude. So a stratification scheme whose cells explain, say, six percent of the variance removes six percent, and the remaining ninety-four percent is exactly where the noise lives. This also explains the shape of the tradeoff around cell count. Adding cells can only increase the variance explained by the partition, so it is tempting to slice finely. But thin cells create their own problems: within-cell estimates become noisy, some cells end up with too few users in one arm, and the bookkeeping grows. The practical rule is few, large, meaningfully-different cells. If the between-cell spread is small, no amount of slicing rescues it. ## When stratification is the right tool anyway - **No pre-period metric exists.** A brand-new surface, a newly defined metric, or a population that did not exist before launch leaves nothing to correlate against. Discrete attributes known at assignment are all you have. - **The metric really is group-dominated.** Some metrics - conversion in markets with very different baselines, latency across device classes - genuinely carry a large share of their variance between groups. - **Small experiments where balance matters.** With few users, chance imbalance on an important attribute is likely and awkward to explain. Guaranteeing balance at assignment removes the conversation entirely. - **Legibility.** A stratified design is easy to describe to stakeholders, and per-cell results fall out naturally. ## Requirements and traps The stratifying attribute must be **pre-treatment**, exactly as a CUPED covariate must be. An attribute recomputed from recent behaviour - an activity tier refreshed daily, a churn-risk segment - looks static and is not; if the treatment moves the underlying behaviour, cells stop being fixed and the design's guarantee evaporates. The analysis has to match the design. If you stratified at assignment, the estimate should be formed within cells and combined across them using each cell's share of the population, rather than pooling everything and hoping. Pooling is not catastrophic when cells are balanced, but it leaves part of the intended gain on the table. ## Composing them These are not exclusive. A common arrangement is to stratify assignment on the handful of attributes with genuinely different baselines and then apply a pre-period covariate adjustment at analysis. The gains overlap: to the extent the strata predict the covariate, they are removing the same variance twice, so the combined reduction is less than the sum of the two. Measure the joint effect on historical data rather than adding the numbers. ## The one-line comparison Where a pre-period version of the metric exists, it usually explains far more of the outcome than any coarse grouping, because a user is the best predictor of themselves. Stratification is the tool for the case where that self-prediction is unavailable, or where you want the balance guaranteed at randomisation rather than repaired at analysis.