Which users belong in the denominator of a checkout conversion metric?

## What a denominator commits you to When you write `checkout conversions / eligible users`, the denominator is doing two jobs at once. It is the normaliser that makes arms comparable, and it is the definition of the population your result is about. Getting it wrong breaks either the causal claim or its interpretation, and the two failures look very different. ## The pre-assignment rule Randomisation buys you one guarantee: the two arms are exchangeable *as assigned*. Any filter applied to the denominator that uses information generated after assignment can destroy that guarantee, because the treatment may change who passes the filter. Concretely, suppose the metric is conversions divided by users who have a payment method on file. Two versions: - **Snapshot at assignment.** You record, for each user, whether a card was on file at the moment they entered the experiment, and you never update it. The filter uses a pre-treatment fact. Inside the filtered set, assignment is still random, so the comparison is still causal. It answers a narrower question — the effect among users who already had a card saved — but it answers it correctly. - **Evaluated at analysis time.** You filter on whether the user has a card on file when you build the table. If the variant nudges people to save a card, the treatment arm's denominator now contains a group of users the control arm's denominator never gets. You are comparing card-savers-under-treatment with card-savers-under-control, which are different populations. The difference you measure is part effect and part selection, and no amount of extra sample fixes it. The general principle is that a legitimate denominator filter is a function of pre-randomisation state only. Applying the same rule symmetrically to both arms is not enough: symmetric rules on post-treatment variables still select asymmetric populations, because the treatment changed the variable. ## Sensitivity versus estimand The reason teams narrow denominators is sensitivity. Users who could never plausibly convert — someone who bounced on the landing page, someone in a market where the feature is not shipped — add zeros to both arms. They contribute variance in the denominator and nothing to the signal, so removing them raises the measured effect size and can shorten the test. That gain is real, but it is not free. The narrowed metric estimates a different quantity. A lift of 4% among users with a card on file, who may be 20% of the base, is not a 4% lift on the business. When someone converts the experiment result into a forecast, the population the denominator described has to be carried along with the number, or the forecast is wrong by the ratio of the two populations. This is why a metric definition should always name its population in the metric name itself — `conversion per card-on-file user` rather than `conversion rate`. There is also a subtler asymmetry: narrowing to an eligible population can make an effect look larger while making it *less* representative of what launching would do, because the excluded users are exactly the ones the feature may have been meant to reach. If a feature's whole thesis is bringing in users who do not yet have a payment method, measuring only card-on-file users guarantees you will miss it. ## Practical rules 1. **Snapshot eligibility at assignment** and store it as an attribute of the assignment record, not as a join computed later. This makes the filter reproducible and auditable. 2. **Never filter on anything downstream of the treatment** — not on later behaviour, not on attributes the treatment can edit, not on whether the user was still around at the end. 3. **Name the population in the metric.** Two metrics with the same numerator and different denominators need two names, or someone will compare them. 4. **Freeze the rule before launch.** Changing eligibility mid-flight means the early days and later days are measuring different populations; if you must change it, recompute both arms from raw logs under one rule, or restart. 5. **Report the denominator counts per arm.** If the two arms' denominators differ by more than sampling noise, something is filtering asymmetrically and the result is suspect until explained. 6. **Decide whether the narrow or the broad population is the decision-relevant one.** For a feature aimed at a specific segment, the narrow one is often right; for a launch decision with a company-wide forecast attached, the broad one usually is. Reporting both, clearly labelled, is a reasonable default. ## What a good answer sounds like A strong candidate does not simply say "everyone who was randomised". They say: start from everyone randomised; narrow only on pre-assignment attributes; state the population in the metric's name; check the denominator counts per arm before believing anything; and be explicit that a narrower denominator answers a narrower question that cannot be reused as a company-level lift.