When is replaying a candidate ranker on last month's impression log a trustworthy estimate?

## What replay is trying to do You have a month of impression logs generated by the production ranker, and a candidate ranker you would like to judge without shipping it. **Off-policy replay**, also called counterfactual or off-policy evaluation, asks: what average outcome would the candidate have produced on this same traffic? The machinery is importance sampling. For each logged session you know what the logging policy did and with what probability; you can compute what the candidate would have done and with what probability; you reweight the observed outcome by the ratio of the two. Sessions the candidate would have handled the way production actually did get up-weighted; sessions where it would have done something else get down-weighted toward zero. Note the difference from correcting position bias in labels: there the denominator is the examination propensity of a *slot*, here it is the probability the logging policy chose that *list*. Both are inverse propensity corrections; they answer different questions. ## Condition 1: logged propensities Importance weighting needs the denominator. A deterministic ranker assigns probability one to the list it produced and zero to every other, so the ratio is zero almost everywhere and no candidate that differs at all can be evaluated. The fix is architectural and has to be decided before the log exists: keep a small amount of deliberate randomness in what is served, and write the propensity of each impression into the log next to the outcome. Reconstructing propensities after the fact from a model checkpoint is fragile — features drift, models get rolled back, ties break differently. ## Condition 2: overlap, and why it collapses The support condition says the candidate may only put probability on actions the logging policy could have taken. A candidate listing ranker whose top result for a query is a listing the old ranker never surfaced is asking about a counterfactual the data cannot answer, and importance weighting cannot invent it — it will silently return an estimate dominated by whichever few sessions happened to overlap. This is the practical killer, and it bites hardest in exactly the situation you care about. A candidate that behaves like production is easy to evaluate and not worth shipping. A candidate that reorders aggressively is worth evaluating and nearly impossible to evaluate. On a large catalog the overlap between the candidate's top-1 and the logged top-k is often a small single-digit percentage. ## Condition 3: enough effective data After weighting, the estimate is effectively built from far fewer sessions than the log contains. The standard diagnostic is the **effective sample size**: the squared sum of the weights divided by the sum of their squares. A log of fifty million impressions can carry an effective sample size in the low thousands, and reporting a tight-looking mean off it is how teams talk themselves into bad launches. Look at the weight distribution too — if the top ten weights carry most of the mass, the number is an anecdote. ## Controls, and what they cost - **Clipping the importance weights** at some maximum bounds the variance and biases the estimate toward the logging policy's observed outcome. That direction is worth stating out loud: a clipped replay tends to under-credit a candidate that is genuinely different, which makes a clipped win more believable than a clipped loss. - **Doubly robust estimation** combines importance weighting with a learned model of the outcome, so the estimate stays consistent if either the propensity model or the outcome model is right. It reduces variance materially and is the sensible default, but it inherits the outcome model's extrapolation exactly where the log is thin. - **Restricting the estimand.** Reporting the candidate's performance on the overlapping slice, together with the size of that slice, is more honest than an extrapolated number for the whole population. If the estimate covers 8 percent of queries, say so; the remaining 92 percent is an open question, not a neutral one. ## Condition 4: the world stood still Last month's log carries last month's catalog, seasonality, promotions and user mix. A ranker evaluated on a holiday log and shipped in February is being judged on a different problem. And the log was generated by a ranker that had been shaping user behaviour for months; the candidate would, over time, shift what users see and click, which no single-shot replay can capture. ## How to use it well Treat replay as a **filter, not a verdict**. It is very good at what it is cheap at: killing candidates decisively, catching pathologies, ranking a shortlist of variants, and doing all that in hours rather than weeks. It is weak at certifying a winner, because the candidates most likely to win are the ones with the least support. The standing organisational lesson is that offline evaluability is a property you have to pay for in advance. A team that logs propensities and keeps a modest randomised slice can answer counterfactual questions for years; a team with a fully deterministic serving stack has a log that can only ever describe what already happened.