In a store-level A/B test, why is your sample size the number of stores, not shoppers?

## Why a whole cluster becomes the unit Some treatments cannot be delivered to an individual. A rearranged shelf, a store-wide price, a new curriculum in a classroom, a city-wide delivery fee — the intervention exists at the level of the group. Sometimes the reason is behavioural instead: members of the group talk to each other or share a resource, so treating one member effectively touches the others. In both cases the honest design flips one coin per store, per classroom, per city, and every member inherits that arm. ## What the sample size becomes The rule is simple and often surprising to people used to web experiments: **the number of independent replicates equals the number of units that were randomized.** Sixty stores means sixty draws, however many shoppers walk through them. The reason is correlation within the cluster. Two shoppers in the same store share the arm, but also everything else about that store: the manager's execution of the change, staffing levels, the neighbourhood's income, local weather, a competitor opening down the road. Their outcomes therefore rise and fall together. Averaging 5,000 correlated shoppers gives a very precise estimate of *that store's* mean, but the quantity that limits the experiment is how much store means vary from store to store — and that variation is not reduced at all by measuring more shoppers inside each store. Mechanically: the store mean has an irreducible store-specific component that no amount of within-store sampling removes. Once you have enough shoppers to pin down each store's mean, additional shoppers buy essentially nothing. Additional *stores* buy a full new observation. ## The practical consequences **Small n, large minimum detectable effect.** With 40 or 60 stores, only fairly large effects are detectable. Teams routinely arrive with a plan sized on shopper counts and have to be told the experiment is powered for a 6% lift, not a 0.5% one. **Analyse at the store level.** Compute one number per store — sales per visit, conversion rate, revenue per opening hour — and compare those numbers across arms. That keeps the analysis unit equal to the randomization unit. **Variance reduction is where the power is.** Since n is fixed by the world, the lever is the variance between store means. Two standard moves: use each store's pre-period value as a covariate or as a baseline the outcome is measured against, and randomize within matched pairs or blocks of similar stores (by size, format, region) so the arms are comparable before the change starts. Blocking on a strong predictor of the outcome can cut the required number of stores substantially. **Beware a few dominant clusters.** Store sizes are usually very unequal. One flagship store can swing a store-weighted average; a shopper-weighted average lets it swing even harder. Decide before the run whether the estimand is per store or per shopper, and if it is per shopper, remember the uncertainty still has to respect that only the stores were randomized. **Randomization checks matter more.** With 60 units, chance imbalance on a variable like store size is common, not rare. Checking pre-period balance and, better, randomizing within matched blocks avoids arms that differ before anything happened. ## The failure mode to name The classic error is running a shopper-level significance test on a store-level design. It combines two mistakes — the analysis unit is finer than the randomization unit, and shoppers within a store are correlated — and it produces standard errors that are wrong by a large factor. The result is a confident report that a shelf change lifted sales, from an experiment that could not have detected anything smaller than several percent. ## What a strong answer sounds like "We randomized stores, so n is stores. Shoppers inside a store share the treatment and the store's own conditions, so their outcomes are correlated and extra footfall adds little. I'd analyse one metric per store, size the experiment on the number of stores and the spread of store means, and use pre-period sales for matched-pair randomization to get the detectable effect down."