How should a rolling-origin backtest reflect how often the model is refit in production?

## The backtest simulates a policy, not a model It is easy to think of a backtest as measuring a model. It is more accurate to say it measures a *policy*: a rule for how a model is built, refreshed and served over time. Refit cadence is a first-class part of that policy, alongside the training window and the horizon, and if the backtest and production disagree about it, the backtest is measuring a system nobody is going to run. The usual mismatch runs one way. Backtests refit at every origin because that is the simplest loop to write. Production retrains far less often, because retraining involves compute, a validation step, sometimes human review or an approval gate, and a deployment with its own risk. The consequence is a systematically optimistic report: the backtest never serves a model older than one step, while production routinely serves one that is weeks or months old. ## What "stale" actually means Between refits, two things can move forward independently: - **The data the model consumes.** A deployed forecaster keeps ingesting new observations, so its inputs stay current even when its parameters do not. - **The parameters themselves.** These change only when you refit. Some model families make the distinction sharp: state can be updated with each new observation while the estimated parameters stay fixed until the next scheduled refit. A faithful backtest imitates exactly that — roll the data forward at every origin, refit the parameters only on the cadence you intend to deploy. The cost of staleness is the accuracy you give up by serving parameters estimated from older data. It depends on how fast the process drifts. On a stable series it can be near zero for months; through a demand shift it can be large and concentrated in a short period. ## Measuring the cost rather than guessing it The decision does not have to be intuition. Run the identical set of origins under two cadences — refit at every origin, and refit on the candidate schedule — and compare origin by origin. That gives: - **The average gap**, which is the headline price of the slower cadence. - **The shape of the gap over time**, which is usually more informative. A gap concentrated in the weeks following a structural break tells a different story from a gap spread evenly, and points to event-triggered retraining rather than a faster calendar. - **The worst-origin gap**, which matters when the business cost of a bad forecast is not linear — a stock-out or an overcommitment during peak season is not compensated by an accurate quiet week. Against that, weigh the costs of frequent refitting: compute and orchestration, the human time in any review gate, the deployment risk of a model changing under people who plan against it, and the loss of a stable reference point when forecasts change for reasons users cannot see. ## Making the call A reasonable decision procedure: 1. Measure the staleness cost at two or three candidate cadences on the same origins. 2. If the slowest acceptable cadence loses little on average and little at the worst origin, take it and spend the saved effort elsewhere. 3. If the loss concentrates around regime shifts, keep the slow calendar cadence but add a trigger: retrain when monitored error or input distribution moves beyond a threshold. 4. Whatever you choose, make the backtest use that exact schedule from then on, so the reported accuracy is the accuracy of the thing being deployed. ## Second-order interactions Cadence interacts with the training window. A sliding window plus infrequent refitting compounds staleness — the model is both forgetting old data and not learning new data. An expanding window plus infrequent refitting is more inert but drifts more slowly. It also interacts with organisational rhythm. If forecasts feed a quarterly planning cycle, a model that changes weekly may create more confusion than the accuracy is worth, because plans get revised for reasons stakeholders cannot attribute. Conversely, a short-horizon operational forecast that drives daily replenishment can tolerate and benefit from frequent refreshes. ## Reporting When you publish backtest accuracy, state the refit cadence alongside the horizon, the window policy and the number of origins. Those four facts define what was actually measured, and omitting the cadence is how an optimistic number reaches a planning meeting without anyone noticing the assumption behind it. ## Interview framing The strong answer treats cadence as a policy choice with a measurable price and a set of costs on the other side, insists that the backtest simulate whatever cadence is chosen, and mentions where the gap concentrates rather than only its average. A weak answer says "refit as often as possible" and never connects the backtest loop to how the system will actually run.