skip to content

In a priority inbox, how would you decide what share of promotion slots to spend on randomised exploration?

level: principalimportance: should knowfreq 38%

answer

  1. price both sides, then argue
  2. cost is share times the gap
  3. size upward from positives needed
  4. express the ceiling per recipient
  5. standing decision with a review date

basics

~20 s

Price both sides and argue the number. The cost is the share times the open-rate gap between a scored and a random promotion, paid today; the benefit is enough uncensored exposures per refresh for suppressed sender classes to be learnable and measurable at all.

solid answer

~50 s

Make it arithmetic rather than taste. Cost: with 45% of scored promotions opened and 6% of random ones, a 2% share moves the promoted view's open rate to 44.22% — 0.78 percentage points, which is the share times the 39-point gap. Benefit floor: decide how many exploration positives a suppressed class needs to be learnable next refresh, and divide by its open rate on random exposure — 5,000 positives at 6% needs roughly 83,000 exposures inside the refresh window. Ceiling: what a recipient actually feels, which for 8 promotions a day at 2% is about one random promotion every six days. Then pick the smallest share that clears the floor, state who bears the cost, and set a date to revisit it — the share that re-seeds a collapsed class is larger than the share that merely keeps the measurement readable.

code

pseudocode · 22 lines
pseudocode
messages_per_day   = 20_000_000
promoted_share     = 0.08
promotions_per_day = messages_per_day * promoted_share      // 1_600_000

explore_share = 0.02
explore_slots = promotions_per_day * explore_share          // 32_000 / day

open_rate_scored = 0.45
open_rate_random = 0.06
blended = (1 - explore_share) * open_rate_scored
        + explore_share       * open_rate_random            // 0.4422
cost_pp = (open_rate_scored - blended) * 100                // 0.78 pp

// does it clear the learning floor for one suppressed class?
exposures_per_refresh = explore_slots * 30                  // 960_000
class_share           = 0.10
class_positives = exposures_per_refresh * class_share * open_rate_random
                                                            // 5_760
if class_positives >= POSITIVES_NEEDED:                     // 5_000
    accept(explore_share)
else:
    raise_share_or_stratify(class_share)

go deeper

for a junior

Recall that showing some mail at random costs quality now and buys data later; it is a trade somebody has to price rather than a switch to flip.

for a middle

Be able to compute both sides: the blended open rate at a given share, and how many exposures a class needs before its positives are enough to learn from.

for a senior

Size upward from the learning requirement, displace marginal promotions rather than adding them, and check the slice is thick enough for the classes most likely to collapse.

for a principal

Own the judgment end to end — who bears the cost, how it is expressed to the product owner, when it is revisited, and what a 0% share commits the team to.

## The shape of the decision This is a genuine business judgment, not a tuning parameter with a right answer. The inbox surrenders some quality in the prominent view today to keep its training data and its measurements usable later. Both sides can be priced, and a design round wants the numbers, not a shrug. ## The cost, worked Take a service delivering **20 million messages a day**, promoting **8%** of them — **1.6 million promotions a day**. Suppose scored promotions are opened **45%** of the time and randomly chosen ones **6%** of the time, a gap of **39 percentage points**. The blended open rate on the promoted view at an exploration share `s` is `(1 - s) x 0.45 + s x 0.06`, so the cost is simply `s x 39` percentage points: | Exploration share | Random promotions / day | Promoted-view open rate | Cost | Exploration exposures / 30-day refresh | |---|---|---|---|---| | 0% | 0 | 45.00% | none | 0 | | 1% | 16,000 | 44.61% | 0.39 pp | 480,000 | | 2% | 32,000 | 44.22% | 0.78 pp | 960,000 | | 5% | 80,000 | 43.05% | 1.95 pp | 2,400,000 | At 2% that is a 1.7% relative reduction in the promoted view's open rate — small enough to be argued for, large enough that somebody owns the number. ## The floor: what the next refresh actually needs Size upward from the learning requirement, not downward from what feels affordable: 1. Decide how many positives a suppressed sender class needs in the next training set to be learnable — say **5,000**. 2. Divide by that class's open rate on random exposure: `5,000 / 0.06 = ~83,000` exposures needed inside the refresh window. 3. Check the slice delivers it. A 2% share gives **960,000** exploration exposures per 30-day refresh; a class receiving 10% of them gets **96,000** exposures and about **5,760** positives. It clears. 4. If a class you care about would get far less than that, either raise the share or allocate part of the slice to that class specifically. A slice sized only to keep the *measurement* readable is much smaller — a few thousand exposures per class per refresh is enough to compare rates, though not to refit on. ## The ceiling: what a recipient feels A recipient receiving **100 messages a day** sees about **8** promotions. At a 2% share that is **0.16** random promotions a day — roughly **one every six days**. That number, not the aggregate percentage point, is what the product owner is really being asked to approve. Expressing the cost per recipient rather than per fleet is usually what turns the argument from abstract to decidable. ## Who bears it, and how to make that fair The same total share can be spread thinly across all recipients or concentrated on a few. Thin and universal is the usual choice, because a concentrated slice gives a small group a visibly worse inbox and the rest a free ride on their data. Where the mail carries real consequence, an unconditional random promotion may be the wrong instrument anyway — capping which classes are eligible for random promotion is a legitimate constraint that costs some coverage. ## Revisit it, do not set it The right share is not constant: - After a class's coverage recovers, the share sized to re-seed it is larger than the share needed to keep watching it. - After a big change to the ranker or to the sender mix, the censoring is fresh and the share should go back up for a while. - If the randomised slice is too thin to read for the classes most at risk, the measurement has quietly stopped working while still appearing in the report. Book a review each refresh cycle and treat the share as a standing decision with an owner. ## Spending nothing is also a decision A 0% share is defensible for a system where a wrong promotion is expensive and the catalogue of senders is small and stable. What it commits to must be said out loud: the training set stays censored by the live ranker, the offline numbers stay unreadable as evidence about suppressed classes, and the first honest signal of a collapse will be recipients complaining. That is a trade a lead can make deliberately; it is not a trade to make by omission.

  • The share is set at 2% and never revisited. What goes wrong?
    Two different failures, in either direction. If the sender mix or the ranker changes, the censoring intensifies and 2% may no longer deliver enough exposures for the classes most at risk, so the control quietly stops being readable while still appearing in the report. If coverage has recovered, the same 2% keeps paying re-seeding prices for measurement-grade needs. Give the share an owner and a review each cycle.
  • How do you argue this share to a product owner who sees only the open rate?
    Convert both sides into what a recipient experiences. The cost is about one randomly promoted message every six days for someone getting eight promotions a day, and 0.78 percentage points off the promoted view's open rate. The benefit is that mail the ranker has written off can still come back, and that the monthly quality report means something. Numbers per recipient decide this conversation; fleet percentages do not.
  • Is a 0% exploration share ever the right answer?
    Yes, where a wrong promotion is costly and the sender population is small and stable enough that the ranker is unlikely to lose a whole class. The commitment has to be explicit: the training set stays censored by the live ranker, offline numbers carry no evidence about suppressed classes, and the first reliable signal of a collapse will be a complaint. Choose it deliberately, not by never raising the question.

saying these in an interview costs you the question

  • Picks a percentage by feel with no cost computed
  • Treats the cost as the full open-rate gap rather than share times gap
  • Sizes the share downward from budget instead of upward from positives needed
  • Concentrates all exploration on a small set of recipients
  • Sets the share once and never reviews it
  • Argues the cost in fleet percentages the product owner cannot feel