When would you randomize an A/B test by session rather than by user?

## The choice User randomization assigns a person once and keeps them in that arm for the life of the experiment. Session randomization assigns each visit independently, so the same person can move between arms across visits. Both are legitimate designs; they answer different questions and carry different risks. ## The case for sessions **More units.** A population of 100,000 people may produce 400,000 visits in the same period. Because the precision of a comparison improves roughly with the square root of the number of independent units, a valid session-level design can detect a smaller difference in the same calendar time — but only if the sessions really are close to independent, which they are not when a small group of people supplies most of the visits. **Tighter scoping.** If the change only exists inside a visit — a banner, a sort order, a page layout — and the outcome is completed inside the same visit, the session is the natural experimental frame. **Traffic mix.** Session-level splits balance day-of-week and campaign mix continuously, so both arms see similar traffic composition even if the population shifts. ## Carryover: the reason to prefer users Carryover means an earlier exposure changes later behaviour. It quietly destroys session designs. Take a session-randomized onboarding tutorial. On visit 1, a person is assigned treatment and completes the tutorial. On visit 2 they are re-randomized into control, and they navigate the product fluently because they were taught last time. Their control-arm session now looks good for reasons that belong to the treatment. Aggregated over many returning people, the control arm absorbs a share of the treatment's benefit and the measured gap shrinks toward zero. Worse, the contamination grows over the life of the experiment, so the effect appears to fade — a pattern that is easy to misread as novelty wearing off. Carryover comes in many forms: learning (the tutorial), habit (a new default the person now expects), state (a saved setting or a completed profile), and annoyance (a nagging prompt they already dismissed). Any of these makes session randomization invalid. ## What metrics you can no longer define A person-level metric requires the person to belong to exactly one arm. Retention ("did this user return within 7 days"), lifetime value, subscription conversion and complaint rate are all person-level. If a user visited three times with two treatment sessions and one control session, there is no honest way to place them in an arm, and any rule you pick induces bias. Session randomization therefore restricts you to session-scoped outcomes: same-visit conversion, visit length, task completion within the visit. ## Independence, and the analysis unit Even when carryover is genuinely absent, sessions from the same person are correlated: heavy users contribute many sessions with similar behaviour. Treating each session as an independent observation understates the true variability of the comparison and produces confidence intervals that are too narrow. If the same person can appear in both arms, that correlation also links the two arms, which is not something a simple two-sample comparison expects. The safe habit is to make the analysis unit match the randomization unit; when it cannot, the correlation has to be handled explicitly rather than ignored. ## The inconsistency people can see Session designs also create visible inconsistency: someone who liked yesterday's checkout finds today's different. For small cosmetic changes this is tolerable; for anything a person invests effort in, it is a product defect, and support tickets about "the site keeps changing" are a real cost. ## A decision procedure 1. Can an earlier exposure change later behaviour? If yes, randomize by user. 2. Is the primary metric defined over a person rather than a visit? If yes, randomize by user. 3. Would a returning person notice the switch and be bothered? If yes, randomize by user. 4. Otherwise session randomization is available, and its extra units are a genuine gain — provided the analysis respects that one person's sessions are correlated. ## What good candidates add Strong answers point out that the choice is not just statistical hygiene: user and session randomization estimate different things. A user-level design estimates the effect of *being the kind of person who always gets this experience*, which is what shipping actually does. A session-level design estimates the effect of *this visit having the feature*, which is a narrower and sometimes less decision-relevant quantity.