In scikit-learn, why must a scaler live inside the Pipeline passed to cross_val_score?
answer
- who calls fit, and on which rows
- the scaler already saw the held-out rows
- refit per fold, not once up front
- Pipeline.fit chains fit_transform down the steps
- transform only on the validation fold
basics
~20 scross_val_score refits whatever estimator you hand it on each training fold. A scaler fitted outside it has already seen every fold's held-out rows, so its means and variances encode validation data and the reported score comes out optimistically biased.
solid answer
~40 s`cross_val_score` clones the estimator you pass and refits that clone on each training fold, then scores it on the held-out fold. If you call `StandardScaler().fit_transform(X)` first and pass the transformed array, the scaler's mean and scale were computed over all rows — including the rows that will be held out — so information from the validation fold is baked into the training features. The score you print is then not an estimate of unseen-data performance. Wrapping the steps in a `Pipeline` (or `make_pipeline`) makes the whole chain the estimator: inside each fold scikit-learn calls `fit_transform` on every transformer using only the training rows, then `transform` only on the held-out rows before `predict`. The same object then behaves identically at serving time, because refitting and applying are one contract instead of two scripts.
go deeper
Know the rule and say it plainly: preprocessing goes inside the Pipeline you hand to cross-validation, never fitted on the full dataset beforehand. Be able to write the two-line make_pipeline version on a whiteboard.
Explain the mechanics — cross-validation clones and refits the estimator per fold, so a transformer fitted outside is never refitted, and its statistics carry validation rows. Say what Pipeline.fit versus Pipeline.predict calls on each step.
Show you can find this in someone else's code and quantify it: which transformers hold learned state, why feature selection and target-based encodings leak hardest, and how leaked scores distort model comparison rather than just adding noise.
Own it as a process problem. Argue for the pipeline as the unit that crosses from evaluation into serving, and for review or CI checks that catch a transformer fitted outside the searched estimator, because a leaked benchmark propagates into every downstream decision.
## The setup A typical modelling script has two phases that look independent: preprocess the feature matrix, then evaluate a model on it. In scikit-learn those phases are two different API calls — `transformer.fit_transform(X)` and `cross_val_score(estimator, X, y)` — and nothing in the library forces them to agree about which rows are "training" rows. That gap is where leakage enters. ## What cross_val_score actually does `cross_val_score(estimator, X, y, cv=5)` does not fit `estimator` once. For each of the five splits it calls `clone(estimator)` to get a fresh, unfitted copy with the same hyperparameters, fits that copy on the training indices only, and scores it on the validation indices. The point of cloning is that each fold's model must know nothing about the rows it is scored on. That guarantee covers exactly one object: the estimator you passed. Anything you did to `X` *before* the call is outside the guarantee. If `X` is the output of `StandardScaler().fit_transform(X_raw)`, then every column was centred and scaled using statistics computed over all n rows. Fold 3's validation rows contributed to the mean that fold 3's training rows were shifted by. The model is trained on features that were computed with knowledge of its own test set. ## Why the bias is systematic, not just noise A common wrong answer is that this "only adds a little noise". It does not — it biases the estimate in one direction. Preprocessing fitted on the full data makes the training and validation distributions artificially better aligned than they will be in production, where you genuinely cannot see future rows when you fit the scaler. The magnitude depends on the transform: for `StandardScaler` on a large, well-behaved dataset it may be a fraction of a point; for a `SelectKBest` feature selection or a `TfidfVectorizer` vocabulary fitted on everything, or on a small dataset, it can be several points of accuracy — enough to pick the wrong model. The rank ordering is what really breaks. Leakage does not inflate every candidate equally, so a search that compares models on leaked scores can prefer the model that exploits the leak best. ## What Pipeline changes mechanically `Pipeline([("scaler", StandardScaler()), ("clf", LogisticRegression())])` is itself an estimator: it implements `fit`, `predict`, `get_params` and `set_params`, so `clone` copies it whole. Inside a fold: - `Pipeline.fit(X_train, y_train)` calls `fit_transform` on each step except the last, in order, threading each step's output into the next, then `fit` on the final estimator. - `Pipeline.predict(X_val)` calls `transform` — never `fit` — on each intermediate step, then `predict` on the final estimator. So the scaler's mean and variance come from the training fold alone, and the validation rows are only ever transformed by them. Every step is refitted per fold, not just the classifier. `make_pipeline(StandardScaler(), LogisticRegression())` is the same thing with auto-generated step names. ## Which transforms actually leak Useful distinction for an interview: - **Row-independent, stateless** transforms (a log transform, a fixed unit conversion) compute nothing from the data, so fitting them on everything changes no number. They still belong in the pipeline, for train/serve parity — but they are not the leak. - **Transforms with learned state** are the leaky ones: `StandardScaler`/`MinMaxScaler` (means, ranges), `SimpleImputer` (medians), `OneHotEncoder` (the category set), `PCA` (components), `TfidfVectorizer` (vocabulary and document frequencies), and any feature selection, which is the worst offender because selecting the top-k features against all of `y` can turn pure noise into a strong-looking model. - **Transforms that consume `y`** leak hardest. `TargetEncoder` is the notable case where scikit-learn defends you: its `fit_transform` uses internal cross-fitting so the encoding of a row is not computed from that row's own target, while `transform` uses the full-data encoding. That is a property of the estimator, not a substitute for putting it in a pipeline. ## The single-split case Splitting first with `train_test_split`, fitting the scaler on the training half and calling `transform` on the test half is correct for a one-shot holdout. The trouble is that the moment you tune anything with cross-validation *inside* the training half, the inner folds have the same problem again — the scaler saw all of the training half, including each inner validation fold. Putting the transforms in the pipeline solves both layers at once and is the habit interviewers look for. ## How to spot it in review Grep for `fit_transform` or `.fit(` on a transformer applied to the full feature matrix, anywhere above a `cross_val_score`, `cross_validate` or `GridSearchCV` call. If the transformer is not a step of the estimator being searched, the score is not trustworthy.
- If I split with train_test_split first and only transform the test set, am I safe?For a single holdout, yes — the scaler is fitted on training rows and only applies to the test rows. But as soon as you cross-validate or grid-search inside the training half, the scaler has seen every inner validation fold, and the bias returns one level down. Putting the transforms in the pipeline fixes both layers with one change.
- Inside a fold, which steps of the pipeline get refitted — all of them, or just the estimator?All of them. `cross_val_score` clones the entire pipeline, so each fold gets fresh, unfitted copies of every transformer. Each intermediate step runs `fit_transform` on the training fold and `transform` on the validation fold; only then does the final estimator fit. That is why the count of scaler fits equals the number of folds, not one.
- Are there transforms where fitting on the whole dataset genuinely changes nothing?Yes — transforms that learn no state, such as a fixed log or unit conversion wrapped in `FunctionTransformer`. They compute the same output row by row regardless of what else is in the matrix. They still belong inside the pipeline so training and serving share one artifact, but they are not the source of an inflated score.
- How large is the effect in practice?It depends on the transform and the sample size. Full-data standardisation on a large tabular dataset might shift the score by a fraction of a point; a `SelectKBest` or `TfidfVectorizer` fitted on everything, or any transform touching `y`, can move it by several points. The dangerous part is that it inflates candidates unevenly, so model comparisons can flip.
saying these in an interview costs you the question
- Scaling uses no labels, so it cannot leak
- fit_transform on all of X and then splitting is fine
- A Pipeline only saves typing, it doesn't change the score
- Only the final estimator is refitted on each fold
- Leakage just adds noise rather than biasing the estimate