How do you get an organisation to commit to a fixed experiment runtime before launch?

## The problem is organisational, not statistical Every practitioner knows a test should run whole weeks and to a committed end date. Tests still get read early, extended halfway, and squeezed around launches. The pressure is structural: a team has a quarterly commitment, a launch date, a leadership review on Thursday, and a queue of other tests waiting for traffic. A lead's job is to change the structure so the correct behaviour is the easy one. ## Make the default carry the weight - **A standard runtime.** Pick one — two weeks is a common choice because it covers two full cycles — and make it the pre-filled value in the test plan template. Defaults win; the effort should be on the exception, not on compliance. - **Whole multiples of seven, always.** Framing runtimes in weeks rather than days removes an entire category of argument about whether nine or ten days is enough. - **An explicit end date in the plan.** Not just a sample target. A date is checkable by anyone and does not require reading a dashboard to evaluate. - **A named exception path.** Shorter windows are allowed, with a stated criterion: historical evidence that the metric has no weekly pattern, an emergency regression check, a genuinely immovable external deadline. Requiring a written reason is what stops the exception becoming the norm; making exceptions impossible is what drives people to stop writing plans at all. ## Speak in the leadership currency The argument that lands is not about representativeness. It is about the cost of a wrong decision. Shipping a change on a mid-week-only estimate that does not survive contact with a full week means rework, a rollback, or worse, a permanent change that quietly costs money and is never revisited. Set that expected cost against the delay: a few extra days of calendar, once, per test. When the change is reversible and cheap, the honest answer may genuinely be to accept a shorter window and ship — and being willing to say so is what makes the argument credible the rest of the time. ## Remove the pressure at its source Most demands to shorten runtimes are really demands for throughput. Attack that directly: - **Smaller allocations, more concurrency.** If a test only needs a fraction of traffic to hit its sample in two weeks, several tests can run simultaneously and each still gets its full calendar. Throughput rises without any window shrinking. - **A shared launch and calendar view.** Holidays, promotions and other teams' launches are visible in advance, so windows are planned around them rather than defended during analysis. - **Front-load the decisions.** Sample target, runtime, primary metric and the exact decision rule are agreed at plan review. Once those are agreed publicly, mid-flight renegotiation has a social cost. - **A ready answer for "we need to know now".** Offer the honest interim: a directional read with the limitations stated, explicitly labelled as not the decision. This is far better than either refusing to communicate or letting the interim quietly become the decision. ## What you are trading Be candid that the policy has real costs. Fixed windows slow the learning loop, particularly for small teams with low traffic where two weeks may be a meaningful share of a quarter. They can block a launch date. They occasionally hold a clearly beneficial change back for days. The counterweight is a decision record you can trust, which is what makes an experimentation programme worth funding at all. A leader who presents only the upside is not being trusted with the tradeoff. ## How to answer this in an interview There is no single right answer, and the interviewer knows it. What distinguishes a strong response is the shape: a default that makes the right thing free, a written exception path rather than a ban, an argument framed in decision cost rather than statistical virtue, and an attack on the throughput pressure that causes the problem. Naming the cost of your own policy is the part most candidates skip and the part that signals seniority.