Why does a per-city average demand computed over the full three-year history leak, even with forward-chaining folds?

## The leak is in the pipeline, not the split This one is dangerous because the code that reviewers stare at - the fold construction - is correct. Folds are ordered, nothing from the future appears on the training side by date, the boundary assertions pass. The contamination entered earlier, when a feature was built by aggregating across the whole dataset. A per-city average demand computed once over three years of store history and joined onto every row is a single number per city, derived from every day in the file. Attach it to a row dated January 2023 and that row now carries a compressed summary of 2024 and 2025. The row's feature vector contains its own future. ## The two separate failures **Backwards flow of the future into training.** The model learns relationships conditioned on a number that already encodes how the series turned out. If a city's demand climbed steeply in year three, its whole-history average is elevated, and year-one rows for that city carry an elevated feature. The model can pick up "cities whose average is high relative to their current level will grow", which is not a discovered pattern - it is the answer, written into the input. **Validation rows summarising their own outcomes.** The aggregate was computed over the validation period too. A validation row's feature is partly a function of that row's own target and of its neighbours' targets. Predicting a value from a summary that contains it is close to trivial, and the fold score reflects that rather than any forecasting skill. ## Why refitting the feature per fold is necessary but not enough The standard reflex is "recompute the aggregate inside each fold's training data". That closes the second failure - the validation period no longer feeds the number - and it is the right first move. It does not close the first. Within a training block spanning January to December, a single aggregate over the whole block still gives a January row a value influenced by December. That is harmless for the fold's own score, but it trains the model on a quantity that has no runtime counterpart: in production, scoring a row on 3 January, the only thing computable is an average over data up to 2 January. The consequence is a train-serve mismatch rather than an inflated fold score. The model relies on a stable, whole-period statistic; live it receives a noisier, shorter-history one, especially for cities that are new or thinly observed. Performance degrades in ways the offline number never predicted. ## The as-of construction The repair is to define the feature the way production must compute it: for a row dated `d` in city `c`, the value is an aggregate of city `c`'s rows dated strictly before `d`. An expanding average uses everything up to `d`; a trailing window uses a fixed recent stretch. Either way the column becomes reproducible at scoring time and contains no future. Three practical consequences follow. Early rows have few or no prior observations, so the feature needs a defined fallback - a global prior, a shrunk estimate, or an explicit missing indicator - and that fallback must be identical offline and online. The column becomes noisier, particularly at the start of a city's history, and the model's measured performance drops; that drop is the illusion leaving, not a regression. And the computation is heavier, since a single grouped mean becomes a per-row running statistic. ## Shrinkage and small groups When a city has three observations, its as-of average is mostly noise, and a model that leans on it will overfit those cities. Shrinking each city's estimate toward the global mean by an amount that depends on its count - heavy shrinkage when data is thin, almost none when it is plentiful - keeps the feature stable. The shrinkage strength is itself a hyperparameter and belongs inside whatever is refit per fold. ## Spotting the class of bug The question that catches all variants of it: *for each column, at what moment could this value have been computed, and does that moment precede the prediction?* Any statistic taken across the entire table - a mean, a count, a rank, a normalisation constant, a frequency encoding, a cluster assignment - fails that test unless it was explicitly built as-of. A fast empirical smoke test is to train on an early period and score a much later one after rebuilding features from the early period only. If the score collapses relative to the fold estimate, something in the pipeline was quietly reading across the whole timeline. A structural safeguard beats vigilance: build features through a path that only ever accepts data with a timestamp strictly earlier than the row being served. When the same code produces both the training column and the live one, whole-timeline aggregates stop being expressible.