Why does randomizing communities in a social graph, rather than users, reduce spillover bias?

## The idea in one sentence If interference travels along connections, then make the randomized unit big enough that most connections live *inside* it. That is the whole of graph-cluster randomization: partition the social graph into groups of densely connected users, then flip a coin per group rather than per user. ## Why user-level randomization leaks so badly Consider a 50/50 user-level split. For a user with k connections, the expected number of connections in the opposite arm is k/2. Nobody is a clean control and nobody is a clean treatment — every user sits in a blended world. The contrast you estimate is not "everyone treated versus nobody treated", it is "half-exposed users assigned treatment versus half-exposed users assigned control", which is a much weaker contrast and generally a biased proxy for the launch decision. Cluster randomization changes the exposure profile. If the partition is good, a user's connections are overwhelmingly inside their own cluster, hence in their own condition. A treated user is surrounded by treated users; a control user is surrounded by control users. The estimated contrast moves much closer to the global comparison you actually want. ## The cut-edge fraction is the bias knob The useful summary statistic of a partition is the **fraction of edges that cross cluster boundaries**. Every within-cluster edge is safe: both endpoints share a condition, so no cross-condition exposure happens along it. Every cut edge is a potential leak, and it leaks only when the two clusters landed in opposite arms — roughly half the cut edges under a 50/50 assignment. So the residual bias is approximately proportional to the cut-edge fraction. Real social graphs are not cleanly separable — they are small-world, with high-degree users bridging everything — so the cut fraction is never zero. Reporting it is part of reporting the experiment: a design that leaves 30% of edges cut has removed far less contamination than one that leaves 5%. ## What it costs: the power collapse The cost is severe and routinely underestimated. Assignment varies **between clusters only**; every user inside a cluster shares the same treatment, and their outcomes are correlated both because similar people cluster together and because they interact. The precision of the estimate is therefore driven by: 1. **the number of clusters**, not the number of users, and 2. **how much clusters differ from one another** in their aggregate outcome. An experiment with forty million users in three hundred clusters is, statistically, closer to a three-hundred-observation study than to a forty-million-observation one. Reporting a user-level standard error on such a design produces intervals that are wildly too narrow and a false-positive rate far above the nominal level. The clean way to think about it is to aggregate: each cluster contributes one number — its mean outcome per user — and the analysis compares those cluster-level numbers across arms. This is why cluster tests detect only large effects. If the organisation's typical shippable effect is a fraction of a percent, a cluster design may simply be unable to see it, and that is a legitimate reason to keep a biased user-level test for routine changes and reserve the cluster design for changes where interference is the whole question. ## The bias-variance dial: cluster size Cluster size trades the two failures directly against each other: - **Larger clusters** contain more edges internally, cutting fewer, so less bias — but fewer independent units, so less power. - **Smaller clusters** give many units and better power — but cut more edges, so more leakage. The practical recipe is to partition into the smallest clusters whose cut-edge fraction is still acceptable, then check whether the resulting cluster count can detect the effect size you care about. If it cannot, the honest conclusions are to enlarge the population, accept a coarser question, or choose a different design entirely. ## Operational traps - **The partition is an estimate.** Community detection produces one plausible grouping of a graph that has no ground-truth communities. Different algorithms give different partitions and therefore different cut fractions. - **It goes stale.** The graph keeps changing while the experiment runs. Freeze the partition and the assignment before launch; re-partitioning mid-flight breaks the randomization. - **Cluster imbalance.** Clusters differ enormously in size and activity, so a coin flip over a few hundred of them can leave the arms visibly unbalanced. Balance on pre-period cluster-level outcomes at assignment time — for instance by pairing similar clusters and randomizing within pairs — rather than hoping randomization sorts it out. - **Bridge users.** High-degree users sit on many cut edges and carry a disproportionate share of the remaining leakage. It is worth checking whether results change when they are excluded from the outcome, as a robustness check rather than as the headline. ## What a strong answer contains Name the mechanism (edges carry spillover), state where residual bias lives (cut edges), state the cost in the right currency (clusters are the independent units, so power collapses), and show you would size the test off the cluster count and balance clusters before assignment. Candidates who describe the design but still quote a user-level sample size have missed the point that makes it expensive.