Would you use a bandit allocator or a fixed-horizon A/B test for twelve news headlines whose value expires within hours?

## The question behind the question "Bandit or fixed-horizon test?" is really "what am I buying with this traffic — earnings during the run, or a measured effect afterwards?" Adaptive allocation buys earnings: it shifts traffic toward whichever arm currently looks best, so fewer users see a loser. A fixed-horizon test buys measurement: a pre-committed sample size, an allocation that does not depend on outcomes, and therefore an unbiased difference estimate with a valid confidence interval. You cannot maximise both with the same traffic. ## Score the decision on five axes **1. Objective.** Do you need to *serve* the best option, or to *know how much better* it is? Twelve headlines need serving. A pricing change that will ship permanently needs knowing. **2. Number of arms.** A fixed-horizon test with twelve arms means eleven comparisons against a control, or sixty-six pairwise ones. Controlling the family-wise error rate makes each comparison harder, and the required total sample grows roughly with the number of arms if you want each one powered. Adaptive allocation scales far more gracefully: it spends most impressions on the two or three plausible winners and only enough on the rest to rule them out. **3. Feedback latency versus decision horizon.** Adaptive allocation only works if the reward lands fast enough to steer the next allocation update. Headline clicks arrive within seconds against a horizon of hours — a ratio of thousands. If the reward were a subscription that lands next week, the allocator would be steering on censored data. **4. Repeatability.** Headline selection recurs every day, automatically, with new candidates. There is no human decision meeting to feed, so there is no consumer for a confidence interval. One-shot decisions that a person must sign off on are the opposite case. **5. Precision requirement.** Nobody needs to know that headline 7 beat headline 3 by 0.42 points. If someone does need that number — for a model, a forecast, or a business case — the fixed-horizon test is the only design that gives it cleanly. On all five, the headline problem points the same way. ## What the fixed-horizon test would actually cost here Suppose each headline's click rate is around 4% and you would care about a relative difference of 10%. Powering even a single such comparison takes on the order of tens of thousands of impressions per arm; with twelve arms and multiplicity control, the total climbs further. Meanwhile the headline is stale in hours. The test does not fail statistically — it fails *temporally*. Its answer would be perfectly valid and completely worthless, and every impression spent on the eleven losers during that window is pure loss. ## What adaptive allocation gives up The crucial cost is not statistical sophistication — it is that **allocation depends on observed outcomes**, which breaks ordinary inference. Arms that were starved carry few observations, and those observations are the ones that caused the starving. Their sample means understate their true rates. The arm that was exploited was picked for looking best, so its logged rate flatters it. Running a two-sample test on the raw log is invalid, and the numbers should never be reported as measured lift. There are practical repairs, and a mature setup applies them by default: - **An exploration floor.** Guarantee every arm some minimum share of traffic. This bounds how badly any arm can be starved and keeps assignment probabilities away from zero. - **A uniform holdout.** Route a small fixed fraction of traffic by pure random assignment. Analysed alone, that slice is an ordinary randomised experiment with unbiased estimates and valid intervals — a clean, if less precise, measurement running inside the adaptive system. - **Logged assignment probabilities.** If you record the probability with which each impression was assigned its arm, the log can be reweighted afterwards to recover unbiased estimates of each arm's mean. ## When the answer flips Switch to a fixed-horizon test when: the reward metric matures slower than your update cadence; the decision is one-shot and permanent; you need an effect size defensible to finance, legal or a model that consumes it; the arms interfere with each other so serving more of one changes another's performance; or the metric is a guardrail whose regression you must detect rather than a reward you are chasing. ## How to answer this in an interview Commit to a choice, name the two or three properties of the problem that drive it, and then volunteer the cost of your choice and the mitigation. Candidates who say only "bandits are more efficient" are giving a slogan; candidates who say "adaptive allocation, because feedback is seconds and value is hours and I need a winner rather than a number — and I will keep a 5% uniform holdout so I can still learn something honest" are giving an engineering answer.