Conversion per visitor rose but conversion per session fell in the same A/B test — why?

## The arithmetic first Both metrics count the same conversions. Write them over the same visitor base: ``` conversions / sessions = (conversions / visitors) / (sessions / visitors) ``` So the per-session rate is the per-visitor rate divided by visits per visitor. Any treatment that raises visits per visitor faster than it raises conversions per visitor pushes the per-session rate down. A worked case. Control: 10.0% conversion per visitor, 2.00 sessions per visitor, so 0.100 / 2.00 = 5.00% per session. Treatment: 10.3% per visitor (+3%), 2.12 sessions per visitor (+6%), so 0.103 / 2.12 = 4.86% per session, a fall of about 2.8%. More visitors converted, and each individual visit converted less often, and both statements are true simultaneously. ## Why the denominators are not equally trustworthy When assignment happens at the visitor level, the number of visitors in each arm is fixed by the randomiser. Nothing a user does after exposure changes which arm they are in or that they contribute exactly one to the denominator. That is what makes a per-visitor rate a clean causal contrast: the denominator is a design fact. Sessions are different. A session is created by behaviour — the user comes back, or is away long enough for a timeout to close one session and open another. The treatment can change how many sessions each user generates. So the per-session denominator is a *post-treatment* quantity, and comparing rates over it compares two differently-composed populations of visits. If the treatment brings back marginal users for one extra short visit, those extra visits are mostly non-converting by nature, and they land only in the treatment arm's denominator. This is why the per-visitor metric is the defensible headline and the per-session metric is a diagnostic. It is not that per-session numbers are meaningless — visit efficiency is a real thing a product team may care about — but a per-session movement in an experiment cannot be read as "the change made visits worse" without first ruling out that the change simply added visits. ## The instrumentation trap Before accepting any behavioural story, check whether the treatment changed what counts as a session. Sessionisation is a definitional rule, usually something like "a gap of 30 minutes of inactivity ends a session", sometimes combined with a rule about a new referrer or a new day boundary. Any of these can be tripped by the treatment without user behaviour changing at all: - A variant that adds a full-page redirect or a hard reload may be logged as a new session start. - A variant that changes how long users linger between actions may push some visits over the inactivity threshold, splitting one visit into two. - A variant on a different subdomain or with a different tagging setup may reset the session identifier. In each case the treatment arm records more sessions with the same underlying user behaviour, and the per-session rate falls purely as an artefact. The check is cheap: compare sessions per visitor, median session length, and the distribution of inter-event gaps between arms. If the extra sessions are unusually short and start immediately after a previous session ended, suspect sessionisation, not behaviour. ## How to report it The reporting pattern that avoids arguments: 1. **Headline**: conversions per exposed visitor, with its uncertainty. This is the metric whose denominator randomisation controls. 2. **Alongside**: sessions per exposed visitor, explicitly labelled as an outcome, so the reader can see the denominator moved. 3. **Diagnostic**: conversions per session, described as a per-visit efficiency read, with a note that its denominator is affected by the treatment. With those three numbers on the page, the pattern in this question reads as a coherent story rather than a contradiction: the change brought people back more often, each visit is a little less loaded, and the net effect on people converting is positive. ## What decision follows If the per-visitor metric is up and the per-session metric is down solely because visits went up, the change is usually good and the per-session drop is a cost-of-success artefact — unless visits themselves are expensive. If serving visits carries real cost (infrastructure, support contacts, ad inventory dilution), the per-session drop is a genuine efficiency signal that belongs in the decision, but it is an economics argument, not evidence the user experience got worse. The failure mode to avoid is picking whichever denominator makes the result look better after seeing both. Fix the primary denominator before launch, and treat the other as explanation.