How do you choose the headline forecast accuracy metric for a portfolio of thousands of series?

## The question behind the question At portfolio scale the metric stops being a statistical choice and becomes an incentive system. Whatever appears on the weekly slide is what teams will optimise, what budget conversations will hinge on, and what a model refresh will be judged by six months from now. So the design criteria are not only mathematical. ## Criterion one: aggregable across scale Series in a real catalogue differ by orders of magnitude. Any metric averaged across them must be dimensionless, and it must be dimensionless in a way that does not systematically favour one size of series. Averaging per-series percentage error fails badly here. Percentage errors are largest exactly where the actuals are smallest, so the mean across a catalogue is dominated by dormant and low-volume series, and the number moves whenever the long tail is added or pruned. Errors scaled by a benchmark's error avoid this, because the scaling constant is per-series and in the same units as that series. The alternative that keeps percentage legibility is a volume-weighted absolute error: sum the absolute errors across all series and periods, sum the actuals across the same cells, and divide. It is defined whenever total demand is positive, it weights each cell by its size, and it is usually what a stakeholder meant when they asked for percentage accuracy. Its weakness is the mirror image: it is dominated by the largest series, so a small number of high-volume products can hide broad failure in the tail. Neither is correct in the abstract. The right move is to state which weighting the headline uses, and to publish the other one beside it. ## Criterion two: a baseline on the same window An accuracy number alone measures the difficulty of the series as much as the quality of the model. A portfolio of stable weekly series will post a flattering number under any method; a portfolio of promotion-driven, intermittent series will look terrible under a very good one. Comparing this quarter's figure to last quarter's is only meaningful if the mix of series and the volatility of the window are unchanged, which they never are. The fix is to score simple benchmarks on exactly the same held-out window and horizons, and report the model relative to them. Three are worth carrying: - **Naive**: every future period equals the last observed value. - **Seasonal naive**: every future period equals the value one seasonal period earlier — last Tuesday for daily data with a weekly cycle, last December for monthly data with a yearly one. - **Drift**: the last observed value plus the average per-period change over the training sample, extrapolated across the horizon; equivalently, extending the straight line through the first and last training points. A skill score, `1 - error_model / error_baseline`, then reads directly: positive means better than the baseline, 0.2 means the error is 20% smaller, negative means the baseline won. Skill absorbs the difficulty of the window, because a hard quarter inflates both terms. It is common and healthy for the seasonal-naive baseline to beat a carefully tuned model on a meaningful share of series. That is a finding, not an embarrassment. It identifies the series where the machinery is not paying for itself, and routing those to the baseline is often the cheapest accuracy win available. ## Criterion three: the distribution, not just the centre One number cannot describe thousands of series. The mean is dragged by outliers, the median hides them. The two summaries worth putting next to the headline are the share of series where the model loses to its baseline, and a small set of percentiles of the per-series score. Those answer the questions a lead actually has: is this broadly working, and where is it not? Segmenting by lifecycle stage, volume band and intermittency usually explains most of the spread, and each segment often deserves a different treatment rather than a better global model. ## Criterion four: alignment with the decision A symmetric error metric assumes being short and being long cost the same. If the downstream decision is inventory or capacity, they do not, and a portfolio scored on symmetric error will be optimised toward a target the business never wanted. Where the asymmetry is real, the scorecard should carry a quantile-aware score at the operating level, alongside the symmetric one. And if the forecast is consumed as a distribution, calibration and interval width belong on the same scorecard as the point error. ## Criterion five: it must be hard to game and stable over time Any single number becomes a target. Ask, for each candidate, what the cheapest way to improve it without improving decisions would be — dropping hard series from the denominator, forecasting low where the metric is asymmetric, quietly shortening the horizon. Then close those doors: fix the series set, fix the horizons, and define the window rule once so the series and the metric are not both moving. Also keep the definition frozen for long enough that a trend means something. A metric changed every two quarters produces a chart nobody can read, which is worse than an imperfect metric held constant. ## The shape of a good answer Name one headline number, say why it is scale-free, pair it with a same-window baseline and a skill score, add the share of series losing to the baseline, weight by business value, and state which decisions the metric is and is not aligned with. The judgment being tested is whether you understand that the metric will be optimised, and whether you have chosen one where doing so is good for the business.