What does a futility boundary add to a group-sequential experiment, and what does it cost?

## The second reason to stop An efficacy boundary answers one question at each interim analysis: is the effect large enough to stop and declare a win, or a harm? A futility boundary answers the complementary one: is the effect so unpromising that continuing is a waste of traffic? Without a futility rule, a group-sequential design will run to its maximum sample even when the estimate has been flat at zero from the first look. That is expensive: traffic held on an inert change, a decision deferred, and a slot on the surface that another idea could have used. ## How the boundary is derived Two standard constructions. **Conditional power.** At an interim analysis, conditional power is the probability that the final analysis will cross the efficacy boundary, given the data observed so far and an assumption about the true effect going forward. The assumption matters enormously and has to be stated. Common choices: - the effect the experiment was designed to detect (optimistic, gives a lenient futility rule); - the currently observed effect (much more pessimistic once the estimate is near zero); - a value in between, or a range reported as a sensitivity band. A rule such as *stop for futility if conditional power under the designed effect falls below 20%* is typical. **Beta spending.** The mirror of alpha spending. A beta-spending function allocates the total Type II error budget across the looks, and the futility boundary at each look is the value below which the cumulative probability of a false negative reaches the allowed amount. This is the more formal route and slots naturally into a spending-function implementation. ## Binding versus non-binding This is the distinction interviews probe, and it is easy to state backwards. - **Binding futility.** The design assumes the experiment stops whenever the futility boundary is crossed. That assumption removes sample paths that could otherwise have come back and crossed the efficacy boundary, so the efficacy cutoffs can be relaxed slightly while still hitting the nominal error rate. The obligation is real: if you cross the futility boundary and continue anyway, the false-positive rate exceeds the nominal level, because the efficacy boundary was priced on the assumption you would have quit. - **Non-binding futility.** The efficacy boundary is computed ignoring the futility rule entirely, as if the experiment always continues. The futility boundary is then advisory. Overriding it and running to the end costs nothing in false-positive terms; you simply did not take the savings the rule offered. In regulated trials binding boundaries appear where the stopping commitment can genuinely be enforced. In product experimentation, where a team can and will argue to keep an experiment running, non-binding is the sane default: it preserves the guarantee under human behaviour rather than assuming it away. ## The costs, stated honestly - **Power.** Every futility rule kills some experiments that would have crossed the efficacy boundary at full data. A lenient rule costs a little power; an aggressive one costs a lot. If you want to hold the design's power constant after adding futility, the maximum sample has to grow to compensate. - **Assumption sensitivity.** A conditional-power rule computed under the observed effect is far more trigger-happy at an early look than the same rule computed under the designed effect, because early estimates are noisy and often near zero. Reporting which assumption was used is part of reporting the decision. - **Interpretation.** Stopping for futility is not evidence that the effect is zero. It is a statement that this experiment, with this remaining budget, is unlikely to demonstrate the effect it was designed to find. A small real effect may well exist below the resolution of the design, and the write-up should say so rather than record a proven null. - **Timing.** Futility rules at very early looks are the ones most likely to be regretted, since conditional power at a small information fraction is dominated by noise. Many designs deliberately start futility monitoring only after a meaningful share of the information has arrived. ## What a strong answer includes Define the boundary, name conditional power or beta spending as the derivation, get the binding versus non-binding direction right — non-binding may be overridden without inflating false positives, binding may not — and be explicit that the currency spent is power, not the false-positive budget.