Offline recall rose 12% but launch moved enrolments 0% - how should offline results gate launches?

## First: is the divergence real, or did the protocol invent it? Before theorising about user behaviour, audit the measurement. A 12 percent relative offline gain is comfortably inside the range that protocol defects produce on their own: - a split whose held-out events sit at scattered dates, so training carried later activity and the model was rewarded for knowing the future; - a small sampled candidate set, which compresses the tail toward the top and can favour whichever model happens to crowd the head; - an offline exclusion rule that differs from serving, for instance scoring courses the live shelf would never display; - a cutoff larger than the number of slots on the page, so retrieval headroom improved while the visible list did not. Re-score both models under the strictest protocol you can run. If the gap collapses, the story is over and the lesson is about evaluation, not about learners. ## Second: why a genuine offline gain can still move nothing **One label is not the set of right answers.** The held-out enrolment is a single acceptable outcome among many. A model that ranks a *better* course first is scored as wrong, and a model that ranks the *recorded* course first may have earned nothing, because that course is often one the learner was going to find through search or a direct link regardless. **Some targets are unwinnable.** If the held-out course is one the platform had already shown the learner and they declined, then predicted it, no ranking policy can convert it. Offline the item scores as a hit; online it is a course the learner has already rejected. **Attribution and dilution.** If the recommendation shelf drives a modest share of enrolments, even a genuine improvement in shelf quality is a small fraction of the top-line number, and the launch may only be able to bound the effect rather than detect it. Zero measured is not zero; it is an interval whose width you should quote. **Substitution rather than creation.** Better ranking can bring forward an enrolment the learner would have made anyway, moving the timing without changing the total. ## What to change about the gating process 1. **Align the offline metric with the online decision metric.** If the launch is judged on enrolments, the offline target event is an enrolment, at the cutoff the surface shows, on the learners the surface reaches. Every step of distance between the two is a place for the correlation to leak out. 2. **Make offline a screen with a known false-positive rate.** Its job is to decide who gets scarce online traffic. That means a bar high enough to be worth an experiment slot, not a bar that pretends to settle the question. 3. **Keep a launch ledger.** Record, for every shipped change, the offline delta and the online delta with its interval. After a dozen entries you have the empirical relationship for your product, and you can set the offline threshold from data instead of intuition. You may discover it is weak - which is itself the most valuable thing this exercise produces. 4. **Report offline results with uncertainty.** A point estimate over a few thousand held-out learners invites over-reading; a spread across learners makes a 12 percent relative gain look appropriately less certain. 5. **Version the protocol.** One written definition of split rule, candidate set, cutoff, exclusions and target. Numbers from different versions are never compared. Without this, the baseline drifts and the ledger is worthless. 6. **Allow mechanistic exceptions in both directions.** A change with a flat offline number may still be worth traffic when there is a clear causal argument - fresher items, better coverage of the new catalog, lower latency - and a spectacular offline number with no such story deserves more scepticism, not less. ## The over-correction to avoid The wrong conclusion is that offline evaluation is worthless and everything should go to traffic. Online capacity is finite and expensive; if nothing is filtered, the experiment queue becomes the bottleneck and the throughput of the whole team drops. The goal is a calibrated screen whose failure rate you know, not the abolition of screening. ## What this question is really testing Whether the candidate defends the offline number, blames the experiment, or does the mature thing: audit the protocol first, accept that offline and online measure different quantities second, and then change the *process* so the organisation learns the strength of its own proxy instead of relitigating each launch by argument.