Your nav redesign is up for new users and down for returning power users — what do you conclude?

A segment split of this shape is one of the most common readouts a redesign produces, and it is where a lot of experimentation judgment shows up. ## First: the pooled number is not a property of the design If two segments have effects of opposite sign, the overall estimate is their traffic-weighted average. That average is a fact about today's user mix, not about the redesign. A product whose traffic is 30% first-time visitors reports a different overall number from one at 10%, for the same design and the same two segment effects. The average will also drift over time as the mix shifts, which means the same experiment re-run later can flip sign without anything about the change being different. Reporting the single number without the split is the failure mode here. ## Second: this is what change aversion predicts The mechanism is specific. Change aversion is a relearning cost, and a relearning cost can only be paid by someone who already learned something. A user who has never seen the previous navigation has nothing to unlearn and evaluates the layout on its own terms; a user who has opened the same menu in the same corner every day for two years has an established motor habit that the redesign breaks. So the prediction is exactly the observed pattern: positive or neutral for the never-exposed segment, negative for the tenured one, with the deficit largest among the heaviest users. ## Third: a competing explanation fits the same data The honest alternative is that the redesign is worse for the workflows that heavy users actually run. Power users have deeper needs — dense navigation, keyboard paths, quick access to rarely-used destinations — and a design optimised for legibility to newcomers can genuinely degrade those. The cross-sectional split cannot distinguish this from relearning, because both stories predict the same sign in the same segments. ## The discriminating evidence Only the time dimension separates them. Track the returning-user gap against each user's exposure age: - **Shrinking steadily toward zero or beyond**: a relearning cost being paid down. Users are adapting, and the loss is transitional. - **Flat and negative after users have had ample exposure**: not aversion. The design is worse for that segment, and no amount of waiting fixes it. - **Shrinking then stalling well short of parity**: partial adaptation on top of a genuine regression — the most common real-world shape, and the one that most needs to be stated plainly rather than rounded to whichever story the team prefers. A second, weaker signal: split the tenured segment by intensity of prior use. If the deficit scales with how entrenched the prior habit was, that supports the relearning story, since relearning cost should be proportional to what was learned. ## What the new-user segment can and cannot tell you It is tempting to treat the new-user effect as the steady state that everyone eventually converges to, on the grounds that today's new users are tomorrow's tenured users. That is only partly right. New users differ from tenured users in far more than habit: intent, engagement level, the tasks they attempt, and how much of the product they ever touch. Their effect is a useful upper bound on how much of the tenured deficit is habit, not a forecast of where the tenured segment lands. ## The decision, not just the diagnosis Even when the deficit is provably transitional, transitional is not free. Multiply the per-user deficit by the number of affected users and the length of the adaptation period, and it is a real cost paid by the most valuable part of the user base. Worse, that cost is not symmetric in its consequences: a fraction of frustrated power users will not stay to complete the adaptation curve, and for them a temporary effect becomes a permanent one. So a defensible conclusion has three parts: state the two segment effects separately, state which mechanism the exposure-age evidence supports, and state the size of the transition cost you are asking the business to accept in exchange for the long-run position.