Validity Threats

Novelty and primacy are transient reactions to *the fact that something changed*, not to what it changed into. They matter because an experiment measures the difference between arms during the window in which it runs, and if that window is dominated by a transient reaction, the number a team ships on is not the number it will live with. ## Novelty A novelty effect is a temporary lift in the treatment arm caused by newness itself. Users notice that something is different, explore the new surface, click the unfamiliar control to see what it does, and briefly increase whatever engagement metric is being tracked. None of that behaviour reflects durable value; it reflects curiosity. The characteristic signature is an effect that starts large and shrinks as each individual user's exposure accumulates: big on a user's first day of exposure, smaller on their fifth, smaller again on their twentieth. ## Primacy, or change aversion A primacy effect is the mirror image: a temporary depression in the treatment arm caused by disruption of an existing habit. The term in product work is change aversion. Users of a long-established tool have built muscle memory around where things sit. A redesign forces them to relearn the layout, and during that relearning period they are slower and make more mistakes even when the new arrangement is better once learned. The signature is an effect that starts negative and improves with exposure. The special case where the curve keeps climbing past the old baseline, because the new design is genuinely better once mastered, is called a learning effect: a keyboard shortcut that saves time only after users habituate to reaching for it behaves exactly this way. ## Why they are a validity threat It helps to be precise about what is broken. Randomization is intact; the two arms are still comparable; the comparison is an unbiased estimate of the difference *during the observed window*. What fails is the extrapolation from that window to the steady state — a threat to the temporal generalizability of the result, not to the internal comparison. A weak candidate treats novelty as though it were a bucketing or logging failure; it is not. The arms are fine. The horizon is wrong. ## Why the transient sits in one arm Control users see the product they already knew. There is nothing new to explore and no habit to unlearn, so no transient. Treatment users get both the feature and the disruption of receiving it. Because the estimate is treatment minus control, the transient does not cancel — it is added to, or subtracted from, the durable effect. ## Who shows which The two effects usually fall on different populations. Someone who has never used the product before has no prior layout to unlearn, so change aversion has nothing to bite on; they evaluate the new design on its own merits. Heavy, long-tenured users are the opposite: they have the most invested habit and pay the largest relearning cost. That is why a redesign can show a positive effect for people arriving for the first time and a negative one for daily users of the same product, and why the pooled number depends on how those two groups are mixed in the traffic. ## What it looks like in data Both effects are defined on each user's own clock — days since that user first saw the change — not on the calendar. A single-number readout hides them completely. A curve of the treatment-minus-control gap against exposure age reveals them: monotone decay toward a smaller value for novelty, monotone recovery from a deficit for change aversion. ## What it is not A shrinking point estimate is not automatically decay. Estimates wobble, and later exposure buckets carry fewer users and wider intervals, so a lift that falls from +8% to +5% may be nothing but noise. Decay is a claim that needs the pattern to be consistent, to appear for successive groups of users as they enter, and to be large relative to the uncertainty. Equally, a stable effect is not proof of no transient — it can be a novelty boost and a real gain of similar size overlapping in a short window. ## What can be done about it after the fact Three readouts do most of the work. Read the effect restricted to users who have had the change for a while, rather than pooling everyone. Read the segment with no prior habit separately from the tenured segment, since they are answering different questions. And read a long-term holdback, where a slice of users has still never received the feature, to see whether the launch gain is still there months later. Each of these targets the same underlying question: what remains once the reaction to the change itself has worn off.