A substitution ranker's offline gate shuffles a year of logged orders at random. What does a time-ordered split fix?

## What a random shuffle assumes Shuffling a year of logged orders and taking, say, 20% as the held-out data split assumes the rows are interchangeable - that any order is as good a stand-in for the next order as any other. For a substitution ranker that assumption is false in two ways at once, and both of them inflate the gate score. ## The holiday week breaks it Demand regimes shift. In the weeks around a holiday the mix of out-of-stock items changes, shoppers accept substitutes they would refuse in March, basket composition moves, and stores restock on a different rhythm. Under a random shuffle, orders from the *same* holiday hours sit on both sides of the split. The model is therefore fitted on the very regime it is graded in - it has already seen which substitutes work under holiday scarcity when it is asked to predict them. Production never gets that. A model trained on data frozen in November and served in December must extrapolate into the shift. The random split measures interpolation; the job is extrapolation, and the difference is exactly the risk the gate was supposed to price. ## Near-duplicates across the boundary The second failure is finer and survives even without a regime shift. Orders are not independent rows: the same shopper reorders weekly, a store runs out of the same item for three days running, a promotion produces thousands of near-identical baskets in one afternoon. A random shuffle places near-copies of training rows into the scoring window, and a model that memorised the training row scores well on its twin. The gate then rewards recall - the same defect a held-out split was meant to remove, reintroduced by the way the split was drawn. ## Building the split the way the model will be used 1. **Pick a cut** at a real timestamp. Everything at or before it is available for fitting; everything after it is the scoring window. 2. **Leave a gap** equal to the real lag between the training data freeze and the moment the model starts serving. If features and labels are assembled for two weeks before a model version ships, a gate with no gap grades the model on days it will never actually predict fresh. 3. **Score forward**, over a window long enough to contain the regime you care about - if the holiday is the risk, the scoring window must cross it. 4. **Repeat over several cuts** (a rolling origin), so the verdict is not an accident of one unusual week, and report the spread across cuts rather than a single number. 5. **Re-derive features as of each order's timestamp**, so no value computed from later events sneaks into the scoring rows. ## What each split actually measures | | random shuffle | time-ordered split | |---|---|---| | what it estimates | performance on rows drawn from the same days as training | performance on days after the training cut | | regime shift | present on both sides, so invisible | visible, which is the point | | near-duplicate rows | straddle the split and reward memorisation | stay on one side of the cut | | resembles production | no - production never sees the future | yes, including the freeze-to-serve lag | | typical effect on the gate score | optimistic | lower and closer to what launch will show | ## What it costs A time-ordered split is not free. The scoring window is a fixed slice of recent history rather than a resampled 20%, so it is usually smaller, and a quiet period or a single supply incident inside it moves the number. Rolling cuts are the standard answer: several origins, the spread reported alongside the mean, and the **shipping bar** written to account for that spread. You also lose the most recent weeks from training whenever you score forward - a real cost when the freshest data is the most relevant, and the reason some teams gate on a time-ordered window and then retrain on everything up to the freeze before shipping the artifact that actually serves. The design-round answer is short: **the split must mirror the deployment gap.** Anything else measures a task the system is never asked to perform.