What guarantee does an always-valid confidence sequence give that a fixed-horizon confidence interval does not?

## Two different promises The distinction is about **when** the coverage promise applies. A fixed-horizon 95% confidence interval is a procedure attached to one sample size `n`, chosen before data collection. Over repeated runs of the whole experiment, the interval computed at that `n` contains the true parameter `theta` in at least 95% of runs. It says nothing about the intervals you could have computed at other sample sizes along the way, and a rule that stops when the interval happens to exclude a value is not covered by the guarantee. A **confidence sequence** is an entire sequence of intervals `CI_1, CI_2, ...`, one per observation or per batch, satisfying the uniform statement `P(exists t : theta not in CI_t) <= alpha`. The quantifier is inside the probability. In at least 95% of runs, **every** interval in the whole sequence contains `theta`. ## Why the uniform version buys anytime validity Because the promise covers all times at once, it also covers any time you might choose to stop, including a time chosen by looking at the data. Formally, for any stopping rule `tau`, `P(theta not in CI_tau) <= alpha`, since the failure event at `tau` is contained in the event that some interval failed. This is what people mean by anytime-valid or always-valid inference: validity is no longer conditional on having pre-committed to a sample size. ## How they are built The machinery is supermartingale-based rather than central-limit-based. - An **e-value** is a nonnegative statistic whose expectation under the null is at most one. By Markov's inequality, `P(E >= 1/alpha) <= alpha`, so a large e-value is evidence against the null at level alpha, and `1/E` behaves like a conservative p-value. - An **e-process** or test martingale is a sequence of such quantities that keeps this property as data accrue. **Ville's inequality** — the martingale analogue of Markov's — bounds the probability that the process ever reaches `1/alpha` by alpha, for the whole infinite trajectory. - A **mixture sequential probability ratio test** builds one concretely. A plain likelihood ratio needs a specific alternative; the mixture version integrates the likelihood ratio over a distribution across candidate effect sizes, giving a single process valid without committing to one alternative. Inverting it — collecting all parameter values the process never rejects — produces the confidence sequence. A second family is built from **time-uniform boundaries** derived from the law of the iterated logarithm, which is why widths often carry a `sqrt(log log t / t)` term. ## The cost Anytime validity is not free, and quantifying the cost is what separates a real answer from a slogan. - **Width.** At any given sample size the confidence sequence is wider than the fixed-horizon interval for the same data. It has to be: it is protecting against every future look, not one. - **Asymptotic rate.** A fixed-horizon half-width shrinks like `1 / sqrt(n)`. A time-uniform one shrinks like `sqrt(log log n / n)`, an extra slowly-growing factor. The gap narrows in relative terms as data accumulate but never closes. - **Tuning.** With a mixture construction, the mixing distribution is a design knob. Concentrating it near the effect sizes you actually care about tightens the sequence there and loosens it elsewhere; a badly chosen spread wastes power on effects nobody would ship. ## Practical consequences - **Dashboards.** A confidence sequence is the right object to render on a live experiment dashboard, because a reader who checks it daily is doing exactly what it was built to permit. - **Stopping for external reasons.** A launch date moves, a bug forces a shutdown, a holiday truncates traffic. Under a confidence sequence the interval at that moment is valid; under a fixed-horizon design an early truncation forces a judgement call. - **The point estimate is still biased.** Uniform coverage of the interval does not de-bias the centre. If you stopped because the sequence excluded zero, the estimate at that moment is on the favourable side, and shipping decisions based on its magnitude should use a bias-adjusted version. - **It does not license changing the metric.** Anytime validity concerns repeated looks at one pre-specified quantity, not swapping in whichever metric currently looks good. ## Answering it well State the two probability statements side by side, note that the quantifier over time sits inside the probability in one and outside in the other, name the martingale mechanism and Ville's inequality, and be honest about the width cost and the residual estimate bias. Claiming anytime validity is free is the fastest way to lose the point.