Why does a randomly shuffled held-out split overstate a fine-tune's gain?

## The assumption a random shuffle makes A random train/held-out shuffle is only valid when rows are independent draws from the same distribution the model will face in production. Almost no real fine-tuning corpus satisfies that. Support tickets cluster by customer. Clinical notes cluster by clinician and by patient. Agronomy field reports cluster by farm, by region, by season and by pathogen outbreak - a single outbreak generates dozens of near-identical reports within a fortnight. When you shuffle such a corpus, near-twins land on both sides of the boundary. The model sees a report, and at eval time is asked about the report from the neighbouring field two days later with the same symptoms, the same crop and largely the same wording. It answers well. The score is real, but it is measuring how well the model interpolates *inside* the training distribution, and production will not be so kind: production arrives as next season's pathogens, new regions, and phrasings from agronomists who never contributed training data. ## Why fine-tuning makes this worse than it sounds For a base model with no exposure to your corpus, leakage of this kind inflates scores modestly. For a fine-tuned model it inflates them dramatically, because fine-tuning is exactly the process that makes the model good at reproducing the training corpus's specifics. The leaked eval item is the case a memorising model handles best. So the random split flatters the arm you are trying to justify and does not flatter the prompted baseline nearly as much - the comparison is biased, not just noisy. ## Split strategies that hold up **Temporal split.** Train on everything up to a cut-off date; hold out everything after it. This is the closest analogue of deployment, where the model is always predicting the future from the past. It captures drift - new pathogens, new products, changed guidance - which a random split cannot see by construction. Use it whenever the data has a time axis, which is most of the time. **Group or entity split.** Choose a grouping key that reflects the correlation - farm, region, customer, author, case ID, document - and assign whole groups to one side. Every report from Farm 41 is in training or in held-out, never both. This is what stops the model from being credited for recognising an entity it has already met. **Combine them.** Temporal and entity splits catch different leaks: temporal misses a farm that reports in both periods, entity misses drift. A split that is temporal at the boundary and grouped within it is the strongest default. **Near-duplicate removal across the boundary.** Even with grouping, boilerplate and copy-pasted text cross over. A cheap pass - character or token n-gram similarity, or embedding similarity with a threshold - removes held-out items that are too close to any training item. Do this *after* the split, in the direction of removing from the held-out side, so you never quietly enrich training. ## Report the gap, not just the good number The most persuasive way to present this is to evaluate both ways. Run the eval on a random split and on the temporal/entity split, and report both. The difference is a direct measurement of how much of the apparent gain was leakage. A large gap is not a reason to hide the grouped number; it is the finding. It tells you the model's advantage is concentrated in cases resembling training data, which in turn tells you what to collect next. ## Sizing and stratification A grouped split is coarser than a random one, so the effective sample size is the number of *groups*, not rows. Three hundred held-out reports from four farms is closer to four independent observations than three hundred, and the confidence interval must reflect that - cluster your bootstrap by group. If you have too few groups to measure anything, that is a data-collection problem, not a reason to shuffle. Stratify so the held-out set retains the slices you care about: rare-but-severe cases, unfamiliar pathogens, short reports, long reports. Rare classes disappear from a small grouped split unless you deliberately keep them, and rare-and-severe is usually where the ship decision actually lives. ## The mindset The question to ask about any split is: *what correlation between training and eval will production break?* Then break it deliberately in the split. Time, entity and source are the three that break most often. A split built this way will produce a smaller headline number than a shuffle, and that smaller number is the one worth trusting - it is the only one that has ever predicted how a fine-tune behaves on data it was not built from.