Why does oversampling the minority class before the train/test split inflate the validation score?

## The setup A fraud or churn dataset arrives at 2% positives. A common reflex is to balance it — duplicate minority rows, or generate synthetic minority points — and then split into train and test. Cross-validated recall comes back at 0.95, everyone is delighted, and production recall is a third of that. The mistake is one line of ordering. ## Failure one: the same row on both sides Random oversampling literally copies minority rows. If the copies are made before the split, the shuffle scatters them: copy A goes to training, copy B goes to validation. The model sees copy A during fitting, and then is asked to classify copy B, which is byte-for-byte identical. Any model with enough capacity gets it right by memory. Because the minority class is exactly the class you duplicated, the inflation lands squarely on minority recall — the metric everyone is watching. Synthetic oversampling does not escape this. A synthetic minority point is constructed from real minority rows; if one of its parents ends up in validation, the training set contains a point sitting essentially on top of a validation row. The generated point was not in the original data, but the information in it was. Undersampling the majority class before splitting behaves differently: it creates no duplicates, so it does not produce this memorisation leak. It still distorts the split in the second way below, and it discards rows that should have been available for evaluation. ## Failure two: the validation set no longer looks like production Even with no duplicate crossing the boundary, balancing before the split changes the class prevalence of the held-out data. Prevalence-sensitive metrics then report numbers that cannot transfer: - **Precision** depends directly on the base rate. At 50% positives a model looks precise; at 2% the same model's positive predictions are mostly false alarms. - **Accuracy** becomes meaningless in the other direction — the trivial majority baseline moves from 98% to 50%. - **A chosen threshold** tuned on balanced validation data will fire far too often on real traffic. Recall and ROC-AUC are insensitive to prevalence, which is one reason a balanced-validation setup can look internally consistent while precision collapses on deployment. ## The leakage-safe order 1. Split first, stratified on the label so both sides keep the natural prevalence. 2. Inside the cross-validation loop, after the fold split, resample **only** that fold's training portion. 3. Leave the validation fold and the final test set exactly as they came — imbalanced. 4. Report metrics at the real prevalence, and pick the operating threshold on data at that prevalence. Resampling is a fitted step like any other: it is part of the training procedure, not part of the data preparation that precedes evaluation. ## What the honest numbers look like When the order is fixed, the minority recall usually falls and precision becomes interpretable. That drop is not a regression — the earlier number was measuring memorised duplicates. A quick diagnostic when you inherit a suspicious result: check whether any validation row is an exact duplicate of a training row, and check the class balance of the validation set against the raw table. A balanced validation split in an imbalanced problem is a strong smell all by itself. ## Spotting it in a review Read top to bottom and find the line that changes the number of rows. If it appears before the split, or before the fold loop, the evaluation is compromised. The same rule catches other row-changing steps in the wrong place: de-duplicating, dropping outliers, or filtering rows by a data-derived rule. ## What to say in an interview Name both failures — duplicated information crossing the split, and a validation set whose class balance no longer matches deployment — then give the order: stratified split, resample the training fold only, evaluate on natural prevalence. Mentioning that undersampling avoids the first failure but not the second shows you understand the mechanism rather than reciting a rule.