How do you check whether an A/B test effect is decaying over exposure time?

Decay is a claim about how the effect behaves as an individual user accumulates exposure to the change. Diagnosing it is mostly a matter of choosing the right time axis and then not fooling yourself with the composition of each bucket. ## Why the pooled number is blind to it A single treatment-minus-control estimate over the whole run averages every user's first day with their thirtieth. A +8% first-day reaction and a +1% steady state pool to something in between, and that in-between number is reported with a comfortable interval around it. Nothing in the summary reveals that the two components exist. ## Why the calendar axis is the wrong axis The obvious next step is to plot the daily effect against the calendar. This is better than nothing but is confounded by composition. On any given calendar day the treatment arm contains a mixture of users at every stage of exposure: people who joined at launch and have adapted, and people arriving today who are seeing the change for the first time. As the test runs, the mixture keeps shifting, so a downward calendar trend can be produced by changing composition rather than by any individual's effect changing at all. The converse also holds: an effect that decays sharply per user can look almost flat on the calendar if fresh users keep arriving and keep contributing fresh first-day reactions. ## The exposure clock Re-index the data. For every user record the date on which they were first exposed to their assigned experience, then compute, for each integer exposure age d, the metric among treatment users at age d and among control users at age d, and take the difference. Two details make this work: - **The control arm needs the same clock.** Engagement falls naturally in the days after anyone's first visit, in both arms. Plotting only the treatment metric against exposure age will always slope downward, and reading that as decay is a common error. The quantity of interest is the *gap between arms* at equal exposure age. - **Later buckets are thinner and more selected.** Only users who keep coming back reach exposure age 30, and they are systematically the more engaged ones. That makes deep buckets both noisier and drawn from a different population, so a difference between age 1 and age 30 is partly a difference between two groups of people. Restricting to users who have had at least the full window available to them makes the buckets comparable, at the price of dropping recent arrivals. ## Cohorting by first-exposure date The strongest version of the diagnosis splits users into groups by the date each user was first exposed, and draws one exposure-age curve per group. This separates two very different stories that the pooled calendar view confuses: - If every group shows the same shape on its own clock — a large gap early, shrinking to a smaller one — and each group's shape is simply shifted in calendar time relative to the last, that is decay of the per-user effect. It is a property of exposure, and it will repeat for every user who ever gets the feature. - If all groups move together on the same calendar days, regardless of how long each has been exposed, the driver is external to the experiment: a marketing push, an outage, a holiday, a change shipped elsewhere in the product. That is not decay, and it will not repeat per user. - If the earliest group shows a big effect and later groups never do, the change did something one-off at launch, or the population arriving later differs from the population that was already there. ## Distinguishing decay from noise A sequence of point estimates that happens to descend is not decay. Each bucket has its own uncertainty, and the later ones are wide because they hold fewer users. Before calling decay, check that the drop is large relative to those intervals, that it is monotone rather than a single low bucket, and above all that it reproduces across independent groups of users. Reproducing across cohorts is the most convincing evidence available, because each cohort is a separate set of people generating the same shape. ## Reading the direction The same machinery diagnoses the opposite transient. A gap that begins negative and climbs toward zero as exposure accumulates is change aversion wearing off; if it climbs past zero and settles positive, the change carried a real benefit that only materialised once users habituated to it. The decision the readout supports is different in each case, but the diagnostic — exposure clock, arm-to-arm gap within bucket, repeat across cohorts — is identical.