Your org pilots every programme on its worst-performing segment and reports the rebound as impact — how do you fix that?

## Diagnose the failure at the system level The problem is not that one analysis was sloppy. It is that the organisation has a standing rule — pick the bottom decile, run the programme, compare after to before — which manufactures a positive-looking result independent of whether the programme does anything. Under that rule the evaluation has almost no diagnostic power: an excellent programme and a worthless one both produce a rebound, so budget, promotion and roadmap decisions are being made on a number that does not vary with the thing it is supposed to measure. That is the argument to make to leadership, and it is a stronger one than 'the statistics are wrong', because it lands as a capital-allocation problem. The magnitude is estimable, which is what makes the case concrete rather than academic. Take untreated units, correlate the metric in one period with the same metric in the next, and you have the shrinkage factor. Multiply the treated group's selected deviation from the mean by that factor and you have the level the group is expected to reach with no programme at all. If your last three programmes all reported gains smaller than that figure, you have evidence that the pipeline has been booking artifact as impact. ## Fix the counterfactual, not the targeting A common wrong turn is to propose that programmes stop targeting the worst segment. Usually that targeting is commercially correct — that is where the recoverable value sits — and proposing to change it will get the whole recommendation rejected. Say explicitly that who receives the programme does not change; only how it is judged does. The options, strongest first: **Randomise within the eligible pool.** Almost always more units qualify as 'bottom segment' than can be served at once, because capacity is limited. Allocating the first wave at random among qualifying units gives an untreated group that went through exactly the same extreme selection, so both regress identically and the difference is the effect. This costs nothing but sequencing discipline. **Stagger the rollout.** Where randomisation is politically impossible, treat the qualifying units in waves and use the not-yet-treated waves as the comparison for a defined window. Everyone gets the programme; the comparison is temporary. **Multi-period baselines.** Where no untreated comparison exists, ban the selection period as the baseline. Use an average over several prior periods, which contains far less of the specific fluctuation that triggered selection. **Independent remeasurement.** Where possible, do not evaluate on the same noisy instrument used to select. If reps were chosen on last quarter's closed revenue, evaluating on a differently-constructed outcome breaks part of the shared noise term and reduces the artifact. **A published null.** Before launch, require every programme brief to state the expected rebound computed from the metric's period-to-period correlation. This is the single cheapest change: it converts the artifact from an invisible bias into a number on the same slide as the target, and a result that merely matches it is read as no effect. ## Fix the incentives None of this holds if teams are still rewarded for the size of a before-and-after delta on a self-selected group. The measurement convention and the incentive have to move together: evaluation standards belong to a function that does not own the programme's success metric, review templates should require the counterfactual to be named, and 'we targeted the worst segment and it improved' should be treated in review as an unanswered question rather than as a result. ## Know the limits of your own fix Be honest about what the corrections do not solve. A multi-period baseline reduces but does not eliminate the bias, since the selection period usually sits inside the average and the selection still favoured high-variance units. Randomising within the eligible pool answers the effect question for that pool only; it says nothing about units outside it. And no design rescues an evaluation whose outcome window is a single noisy period — insist on enough post-period measurement that a one-step rebound and a sustained change are distinguishable, since the artifact returns a group once to its own level while a real effect keeps holding. ## What the interviewer is testing This is a judgment question with no single right answer, and the strong signals are: separating the targeting decision from the evaluation decision instead of conflating them; proposing something implementable under real political constraints rather than only the textbook ideal; quantifying the expected artifact rather than merely naming it; and recognising that the durable fix is to the standing process and its incentives, not to one analysis. A weak answer stops at 'that's regression to the mean' without saying what the organisation should do differently on Monday.