Your experiment hits its planned sample size in three days. Why keep it running to the planned end date?

## Two preconditions, not one A sample-size calculation answers the question "how many units do I need for this design to detect the effect I care about?" It does not answer "which units?" or "over what stretch of calendar?" A runtime decision answers those. A well-formed test plan therefore states both: a target sample and an end date, with the result read when both are satisfied. A high-traffic product routinely hits the count long before the calendar is done, and that is exactly the situation this question probes. ## Problem one: the days you did not cover Three days of a seven-day cycle is a mid-week sample. If the metric or the effect behaves differently at the weekend — different audience, different intent, different device mix — then the estimate describes mid-week users and is being generalised to everyone. The comparison between arms is still internally sound, because both arms drew from the same three days; what is missing is external validity for the population the feature will actually be launched to. ## Problem two: who arrives first This one is easy to miss and is what distinguishes a strong answer. Exposure happens when a user visits. Someone who visits daily is almost certain to be exposed on day one. Someone who visits monthly has roughly a one-in-thirty chance of appearing in a three-day window. So the early sample is dominated by heavy users, and the tail of light, occasional and lapsed users only fills in as the window lengthens. That matters because heavy users are usually not average users: they convert at different rates, have already learned the interface, and often respond differently to a change. A change that helps newcomers find something may look flat among power users who already knew where it was, and vice versa. So even if there were no weekly cycle at all, a very short window would still measure a different population than a two-week window does. ## Problem three: the system may not have settled The opening days of a launch are also when caches are cold, staged rollouts are still widening, and any instrumentation problem is most likely to be live. Reading a result from exactly that stretch is reading the least stable part of the experiment. ## What to actually do - Keep the test running to the committed end date. State in the plan, before launch, that the read happens when both the sample target and the calendar end are met. - If the sample arrives dramatically early and often, that is a capacity signal. Options: allocate a smaller share of traffic to each test so more tests run concurrently, target a smaller minimum effect that the extra sample can now support, or use the surplus to power secondary metrics and pre-declared segment breakdowns. - Do not convert surplus traffic into an ad-hoc shorter runtime after the fact. A duration chosen after seeing the numbers is not a duration you can defend. ## When a short window is genuinely acceptable There are real cases. A metric with no weekly pattern in historical data, a population that is not visit-frequency-heterogeneous, an emergency guardrail check where a fast read on a catastrophic regression is worth more than representativeness, or a within-session interaction metric that is measured immediately at exposure and has no delayed component. In every one of those cases the argument is made from evidence about the metric and stated in the plan in advance — not improvised because the count filled up quickly. ## How to answer it in an interview Say clearly that sample size and runtime are separate requirements, give the day-of-week reason and the heavy-user reason, and then show judgment: if you are consistently over-supplied with traffic, redesign the experiment portfolio around it rather than shortening windows one test at a time.