An 80% prediction interval covers only 55% of held-out actuals. How do you diagnose it?

## What coverage means A prediction interval carries a nominal level: an 80% interval claims that a future actual falls inside it 80% of the time. Empirical coverage is the check on that claim — over a held-out set of forecast-actual pairs, the share of actuals that landed between the lower and upper bound. Nominal is a promise, empirical is the audit. Coverage of 55% against a nominal 80% is a calibration failure in the dangerous direction. The intervals are too narrow, so every downstream consumer — a safety-stock rule, a capacity plan, an alerting threshold — is being told the future is more predictable than it is. ## Step one: is the gap larger than noise? Coverage is a proportion estimated from a finite number of held-out points, so it has sampling error. If the true coverage were the nominal 80% and you scored k independent points, the standard error of the observed share is `sqrt(0.8 * 0.2 / k)`. With 20 points that is about 0.089, so an observed 55% sits roughly 2.8 standard errors below nominal — unlikely but not impossible. With 200 points the standard error drops to about 0.028 and a 55% reading is unambiguous. There is a second reason to distrust a small count: forecast errors from overlapping windows are not independent. Neighbouring origins share most of their training data and the same underlying shocks, so the effective sample size is smaller than the raw count of points. Treat a coverage estimate from a handful of origins as directional, not decisive. ## Step two: split coverage by horizon This is the highest-yield diagnostic. Uncertainty about a series grows with how far ahead you look, so an honest interval widens with the horizon. If width is roughly constant across horizons — a common symptom when the interval is built from a single residual standard deviation — then coverage will look acceptable at one step and collapse at longer horizons. Pooling hides exactly that. Report coverage as a small table: one row per horizon, the nominal level, the observed share covered, and the number of points behind it. The shape of that table tells you whether you have a uniform width problem or a horizon-scaling problem, and they have different fixes. It is also worth splitting by segment — by series, by season, by regime. Coverage that is fine on stable series and terrible on volatile ones points at a variance model that does not adapt, rather than at a global width that is too small. ## Step three: the usual causes **In-sample residuals.** Interval width is often derived from the spread of residuals on the training data. Those residuals are optimistically small: the model was fitted to minimise them. Out-of-sample errors are systematically larger, so intervals built this way under-cover from the start. **Only one source of uncertainty.** Many interval formulas capture the noise of the process while assuming the model form and its fitted parameters are correct. Parameter uncertainty, model-selection uncertainty and the possibility that the data-generating process shifted are all excluded, and each of them widens the honest interval. **A distributional assumption that does not hold.** Symmetric, Gaussian-shaped intervals under-cover when errors are heavy-tailed or skewed, because the tail mass sits further out than a normal quantile allows. Demand and revenue series are routinely right-skewed. **Heteroscedasticity.** If error size scales with the level of the series or varies by day of week, one global width is too wide in quiet periods and far too narrow in busy ones. Pooled coverage can look mediocre while the busy periods, which are the ones that matter, are catastrophic. **A regime change in the holdout.** If the held-out window contains a shift the training period never saw, no interval built on history will cover it. That is a different finding from a mis-specified width, and the response is a monitoring and retraining question rather than a wider interval. ## Step four: fix and re-measure The most reliable repair is empirical rather than parametric: collect the forecast errors your backtest already produced, group them by horizon, and set the interval bounds at the appropriate empirical quantiles of those errors — the 10th and 90th for an 80% interval. This inherits the true error distribution, including skew and heavy tails, and it widens naturally with horizon because longer-horizon errors are larger. It needs enough errors per horizon to estimate a tail quantile, which is the practical constraint. Then re-measure coverage on data that was not used to set the width. Calibrating and evaluating on the same errors reproduces the original sin of using in-sample residuals. ## Do not stop at coverage Coverage alone is trivially gameable: intervals from minus infinity to plus infinity cover 100% of everything and are useless. Calibration must be reported with sharpness — the average interval width — and the honest summary is a score that trades the two off. An interval score does this by charging the width plus a penalty proportional to how far outside the interval a missed actual landed, scaled up as the nominal level gets tighter. Reporting coverage and mean width side by side is the minimum; two forecasters at identical coverage are ranked by the narrower one.