Which preprocessing belongs inside a persisted scikit-learn Pipeline versus upstream ETL?

## The question behind the question Every team eventually draws a line between "the data platform prepares features" and "the model object transforms features". Put the line in the wrong place and you get one of two failures: training/serving skew, where two implementations of the same transform diverge; or an unmaintainable model artifact that has swallowed business logic it should never have owned. ## The rule: learned state goes inside The test that survives contact with real systems is whether the transform has **parameters estimated from training data**. Inside the pipeline, always: - imputation statistics (a median, a most-frequent category) - scaling parameters (means, variances, ranges, quantiles) - categorical vocabularies — an encoder's known category set, and its policy for unseen values at serve time - text vectoriser vocabularies and document frequencies - learned projections and feature-selection masks - anything using `y`, which additionally must never be fitted outside a cross-validation fold The reason is not tidiness. These numbers *are* part of the model. A scaler mean recomputed at serve time from the serving batch puts inference in different units than training, and the failure is silent — no exception, just worse predictions, often worst on the smallest batches. Upstream, usually: - joins, aggregation windows and entity resolution that need the warehouse - label construction and any time-based cutoff logic - deduplication, hard validity filtering, PII handling - anything expensive that is shared across many models and should be computed once The grey zone is stateless row-wise maths — a log, a ratio, a clip. It leaks nothing wherever it lives. Putting it in a `FunctionTransformer` inside the pipeline buys parity for free; leaving it upstream buys debuggability. Choose per team, but choose once. ## What the in-pipeline choice costs **Pickle fragility.** `joblib.dump(pipe, "model.joblib")` serialises by reference for classes: the file records that a step is `myproject.features.RatioTransformer`, not the class's code. At load time that import path must exist and mean the same thing. Rename the module, restructure the package, or deploy a serving image that does not ship the training package, and the load fails. Practical consequences: define custom transformers in a small, stable, importable module that both training and serving depend on; never define one in a notebook or as a closure; and never pass a lambda to `FunctionTransformer` — module-level named functions pickle, lambdas do not. **Version coupling.** Scikit-learn does not promise pickle compatibility across versions. Since 1.3 it detects the mismatch and raises `InconsistentVersionWarning`, telling you which version wrote the file — a warning, not a guarantee of correctness. So the serving image should pin the exact scikit-learn (and numpy/scipy) versions the artifact was fitted with, and the pin belongs next to the artifact, not in a README. **Contract surface.** Fit on a DataFrame and estimators record `feature_names_in_`; predict on a frame whose columns differ in name or order and scikit-learn complains. Fit on a raw array and you get no such check — column order becomes an undocumented, silently-violable contract. Fitting on named columns is the safer default precisely because it makes the contract explicit, and `set_output(transform="pandas")` keeps names flowing through the intermediate steps for debugging. **Latency and shape.** A transform that is fine over a training matrix may be poor per request. A vectoriser is fast; an expensive aggregation is not. If a step is slow per row, that is an argument for precomputing it upstream and passing it in as a feature — accepting that you now own keeping the two implementations aligned, which is a cost you should name out loud rather than discover. ## Unseen data at serve time This is where the boundary is tested. An encoder meets a category it never saw; a numeric column arrives null when training had none. Those policies must be configured on the estimators inside the pipeline — the encoder's unknown-value handling, an imputer covering columns that were complete in training — because serving code that patches the input before the pipeline is a second implementation of the model's assumptions. Decide the policy at fit time, encode it in the object, and let one artifact answer for it. ## The organisational angle The pipeline boundary is also a team boundary. Everything inside it is owned by whoever trains the model and moves at model-release cadence. Everything outside is owned by the data platform and moves at its own. Learned state inside the artifact keeps the fast-changing, model-specific, leak-prone part in one versioned object that can be rolled back atomically; shared, expensive, cross-model computation upstream keeps it from being recomputed per model. When someone proposes moving a fitted statistic upstream "for performance", the question to ask is who will notice when the two copies of that number diverge — because nothing will raise when they do.