Why does picking features by their correlation with the target before cross-validating inflate the score?

## The textbook demonstration Build a dataset with 20,000 feature columns of pure random noise and a balanced binary label that is also random — by construction there is nothing to learn, and the best achievable error rate is 50%. Now run the sequence that looks perfectly reasonable: 1. Compute each column's correlation with the label, over all rows. 2. Keep the 50 strongest. 3. Cross-validate a classifier on those 50 columns. The reported error comes back in the low single digits — around 3%. Nothing was learned; the number is manufactured entirely by step 1 having been run in the wrong place. ## Why it is so much worse than a leaky scaler Two factors multiply. **It is supervised.** A scaler consumes only the feature distribution; a correlation filter consumes the *labels*, which is precisely the information the evaluation is supposed to withhold. The surviving columns encode a summary of every row's answer. **It searches.** With 20,000 candidates, the maximum sample correlation over pure noise is large by chance alone. Selection keeps exactly the columns whose accidents fit *this* dataset, validation rows included. When the fold is then held out, those accidents are still present in the retained columns, so the model reproduces them and calls it prediction. The more candidates you screen, the bigger the illusion — the optimism grows with the size of the search, not with the size of the signal. ## Why cross-validation does not save you Cross-validation only protects the steps that happen inside it. Freezing the feature set before the loop means every fold's model is built on columns chosen with that fold's labels, so all k estimates are contaminated in the same direction, and averaging them makes the wrong number look stable. A tight standard deviation across folds is not evidence of honesty; it is k repetitions of the same leak. ## The correct placement Selection belongs inside the fold: 1. Split into folds. 2. For each fold, compute the selection rule on the **training portion only** — its correlations, its rankings, its surviving column list. 3. Fit the model on those columns of the training portion. 4. Transform the validation portion with the column list from step 2 and score it. 5. Average across folds. Different folds will now keep different columns, and that instability is itself informative: if the selected set changes wildly, the selection was fitting noise. On the pure-noise dataset above, this procedure returns roughly 50% error, which is the truth. The same rule holds for any selection rule that consults the label — a univariate score, a ranking from a model's coefficients or importances, a wrapper search. Unsupervised filters (dropping constant or near-duplicate columns) leak only the held-out rows' feature distribution and are far milder, but the safe habit is to refit them in-fold too. ## Sizing the damage on a real project When you inherit a suspicious result, two checks settle it quickly. Re-run the whole procedure with selection moved inside the fold and compare the two numbers; the gap is the leak. Or run the original procedure against a randomly shuffled label — an honest pipeline collapses to chance, while a leaky one still returns a flattering score. That second check is the most convincing demonstration to a sceptical colleague, because it needs no theory: the data provably contains no signal, and the workflow still claims one. ## The rule to carry away Anything that consults the target and produces a decision reused at scoring time — which columns to keep, how a category maps to a number, which rows to drop — is part of the model. It is refit whenever the model is refit, and it is measured on data that neither it nor the model has seen. The honest score is always the score of the entire procedure end to end.