When should an experiment's opening days be discarded as a burn-in period?

## What a burn-in is Burn-in is the deliberate exclusion of an experiment's opening window from analysis, on the grounds that the system under test had not reached steady state. It is a statement about infrastructure readiness, not about user behaviour. ## Legitimate reasons - **Cold caches and cold paths.** A new code path serves from an empty cache at first. Latency in the variant can be materially worse on day one for reasons that will never exist in the steady state, and latency drags conversion metrics with it. - **Incomplete rollout.** Deployments propagate across regions, instances or app versions. Until propagation completes, the variant population is not the intended population. - **Components that learn.** A ranking or recommendation stage that trains on logged interactions needs a stretch of the variant's own traffic before its output is representative of what it will do in production. - **Instrumentation being repaired.** A logging or assignment bug found and fixed on day two makes the pre-fix data untrustworthy for both arms. In each case the honest description is: during that window, the thing being measured was not the thing you plan to ship. ## What makes it defensible The difference between a burn-in and a convenient exclusion is entirely in the ordering and the criterion. 1. **Pre-declare it.** The plan names a burn-in length, or names an explicit readiness criterion, before any user is exposed. "We exclude the first 48 hours" is a plan. "We excluded the first 48 hours" written after the read is not. 2. **Make the criterion mechanical.** Cache hit rate above its normal band; rollout at one hundred percent of targeted hosts; the retraining job having completed one full cycle; assignment logs passing their integrity check. None of these mention the experiment's outcome metric. A criterion phrased in terms of the metric — "until the difference stabilises" — makes the exclusion a function of the result and destroys the guarantee. 3. **Symmetry.** The same window is dropped from both arms. Dropping only the variant's bad opening days is not an analysis choice, it is fabrication. 4. **Extend, do not shrink.** If two days are discarded from a 14-day plan, run 16 so the analysed window still covers whole weeks and still meets the sample target. Discarding days without extending quietly turns a balanced two-week design into an unbalanced 12-day one. 5. **Report both.** Show the result with and without the burn-in. If they agree, the exclusion was harmless and the reader is reassured. If they disagree sharply, that difference is itself a finding worth explaining. ## The cleanest alternative Burn-in inside an experiment is a second-best fix. The better move, when you can afford it, is to warm the system before the experiment starts: deploy the code, let caches fill and the rollout complete under a small holdout or a no-op configuration, and only then begin exposure and start the clock. Then there is nothing to discard, no exclusion to justify, and the whole window is analysable. ## How to talk about it An interviewer is usually testing whether you can distinguish a principled, pre-registered exclusion from post-hoc data pruning. Lead with the mechanical causes, state that the criterion must be about system readiness rather than about the metric, mention symmetry across arms, and say that discarded days are added back to the end of the runtime rather than subtracted from it. Mentioning that you would report the analysis both ways is what a careful practitioner adds and what a rushed one leaves out.