In a priority-inbox ranker, what does promoting a small random share of arriving mail give the next refresh?

## Why anything random is needed at all A retraining loop fitted on its own placements can only learn about mail it placed. The fix cannot come from the model, from more of its log, or from a longer history of its log, because all three were produced under the same rule. The only way new evidence enters is if some exposure is decided **without reference to the score**. That is what the randomised promotion slice is: a small, deliberate hole in the ranker's authority over what recipients see. ## The two things it buys They are usually conflated, and they have different sizing arguments: 1. **Uncensored training rows.** A sender class the ranker demotes to near-invisibility still appears in the prominent view occasionally, purely at random. Whatever recipients do with those messages is recorded, so the next refresh sees positives from a class that the closed loop would have reported as dead. 2. **An honest measurement.** Open rate on the randomised slice is an estimate of interest that the current ranker did not shape. It is the control described in any detection scheme for this failure: without it, a class's collapse in the training data cannot be told apart from recipients genuinely losing interest. A slice sized only for measurement can be far smaller than one sized to re-seed learning for a rare class, so name which job you are sizing for. ## Where the slots come from Take the random promotions **out of the existing promotion budget**, not in addition to it — typically by replacing the lowest-scoring promotions in the slate. Two reasons: - **Comparability.** If the promoted volume grows when exploration is switched on, every downstream metric moves for a second reason and the effect of exploration cannot be read. - **Cost control.** The recipient's prominent view has a fixed size in practice; expanding it to make room for random mail spends attention rather than ranking quality, and attention is the scarcer budget. Displacing the *marginal* promotion also keeps the cost at its minimum: the slot surrendered is the one the model was least sure about. ## The logging contract The slice is worthless to the next refresh unless it can be identified later, so at decision time the placement record carries: - a flag saying this promotion was random rather than scored; - the probability the slot was filled that way, since the refresh weights scored and random rows differently; - the model version that produced the slate, so a later analysis knows which ranker's censoring it is looking through. All three are cheap at write time and impossible to reconstruct afterwards — a score can be recomputed from a stored model, but the draw that happened at serving time cannot. ## Design details that decide whether it works - **Re-randomise every period.** Freezing the random choice makes it a fixed extra allowlist, which is another placement rule producing its own censored log. - **Spread it across recipients.** The same total share concentrated on a few recipients buys the same data and hands the whole cost to those few. - **Consider stratifying by sender class.** A slice drawn uniformly over all mail gives the rarest classes almost nothing, which is exactly where the loop bites hardest; allocating some of the slice per class is a pipeline decision, at the price of a more complicated weighting downstream. - **Keep the slice out of the personalisation features it is meant to test.** If a random promotion feeds an engagement feature that raises the class's score next refresh, the control has started to influence the thing it measures. ## What it does not do It does not make the rest of the log unbiased, and it does not repair the months already collected under a closed loop. It gives the next refresh evidence it would otherwise not have, and gives the operators a number they can trust. Whether the quality spent on it is worth those two things is a business call, not an engineering one.