Why must a stacking ensemble's meta-learner be trained on out-of-fold base-model predictions?

## The shape of a stack Stacking (stacked generalisation) is a two-level ensemble. **Level 0** holds several base models — say a regularised ridge regression, a nearest-neighbour model and a gradient-boosted tree ensemble, all trained on the same target. **Level 1** holds a single small model, the *meta-learner*, whose input features are not the raw columns at all but the **predictions of the level-0 models**. The meta-learner's job is to learn how much to trust each base model, and in which region of the input space. That is the whole idea. Everything difficult about stacking is about one question: *which* predictions the meta-learner gets to learn from. ## Why in-sample predictions poison it A model's predictions on rows it was trained on are systematically better than its predictions on new rows. A deep tree ensemble may reproduce its own training labels almost exactly; a nearest-neighbour model with k = 1 will reproduce them perfectly, because each row is its own nearest neighbour. If you feed those in-sample predictions to the meta-learner, it sees three columns that are nearly copies of the target. It learns the only rule that fits: trust the model that is most nearly perfect on training data — which is the model that memorised hardest, not the model that generalises best. Validation on those same kinds of features looks spectacular (a 0.97 AUC on weekly store-sales data is the classic shape of it), and the moment the stack scores rows nobody trained on, the base predictions stop being near-copies of the target and the learned weights are pointed at the wrong model. The score collapses to something at or below the best single base model. ## The out-of-fold construction The fix is to build **out-of-fold (OOF) meta-features**: 1. Split the training set into K folds. 2. For each fold k, train every base model on the other K-1 folds and predict fold k. 3. After K rounds, every training row carries one prediction per base model, and each of those predictions came from a model that never saw that row. 4. Stack those columns into an N-by-M matrix — probabilities rather than hard labels for classification, one column per class per model if it is multiclass — and fit the meta-learner on that matrix against the original targets. 5. **Refit each base model on the whole training set.** The fold models were scaffolding; the full-data models are what score live traffic. The meta-learner is kept exactly as trained on the OOF matrix. At serving time a row goes to all M base models, their outputs become the meta-features, and the meta-learner emits the final prediction. (The folds themselves must respect whatever structure the data has — grouping, time ordering — but that is a general splitting concern, not a stacking-specific one.) One asymmetry is worth naming out loud: the meta-learner is *trained* on predictions from models fitted on (K-1)/K of the data, but *serves* predictions from models fitted on all of it. The full-data models are slightly stronger and slightly differently calibrated than the fold models. This mismatch is usually benign and it shrinks as K grows, which is one reason people use a larger K here than they would for plain model selection, or repeat the OOF procedure with several fold seeds and average. ## What the meta-learner should be Small and heavily regularised. It has only a handful of inputs, those inputs are highly correlated with each other and with the target, and each of them carries fold noise. A regularised logistic or linear model — often with non-negative coefficients, which makes the stack a learned weighted average and keeps it interpretable — is the standard choice. A deep second-level tree ensemble over three correlated columns is a reliable way to fit fold noise and give back the gain. Adding a few raw features alongside the meta-features (so the meta-learner can learn *where* each base model is trustworthy) sometimes helps, and sometimes just gives it more rope. ## Blending: the cheap variant Blending holds out a single slice — 10% is typical — instead of running K folds. Base models train on the other 90%, predict the held-out slice, and the meta-learner is fitted on that one set of predictions. It is one round instead of K, it is much harder to get subtly wrong, and it costs you two things: the meta-learner sees only 10% of the rows, so its weights are noisy, and the base models either lose that slice permanently or must be refitted on everything, reintroducing the train/serve mismatch above. Blending is a reasonable choice under a deadline or on a very large dataset where 10% is still hundreds of thousands of rows; full OOF stacking is the choice when data is scarce. ## Scoring the stack honestly The meta-learner's own OOF score is not a clean estimate of the stack: the meta-learner was fitted using all of those OOF rows, and if you tried several meta-learners you also selected on them. Keep a final hold-out that no level of the stack — no base model, no meta-learner, no threshold choice — has touched, and report that number.