Why does a voting ensemble of three separately tuned gradient-boosted tree models barely beat the best one?

## The gain comes from disagreement, not from headcount Combining models helps only to the extent that they are wrong about **different rows**. Three gradient-boosted tree ensembles trained on the same features with the same objective, differing only in hyperparameters or seed, tend to be wrong about the *same* rows — the ambiguous ones, the mislabelled ones, the ones from a under-represented segment. Averaging three copies of the same mistake reproduces the mistake. For squared error this is exact rather than a metaphor. The **ambiguity decomposition** says that the squared error of a uniformly averaged ensemble equals the average squared error of its members *minus* the average disagreement among members (each member's average squared deviation from the ensemble's own prediction). Disagreement is a subtracted term: it is the entire benefit. Zero disagreement means the ensemble scores exactly what the average member scores. The same intuition carries to classification without the tidy identity. So the lever is not the number of models. It is how differently they fail. ## Measuring it before you build anything Generate out-of-fold predictions for each candidate model — one per training row, from a model that did not see the row — and then look at: - **Pairwise correlation of the out-of-fold predictions**, or better, of the residuals. Two models correlated at 0.99 on residuals will not help each other. - **Disagreement rate** for classifiers: the fraction of rows where two models predict different labels. - **The oracle ceiling**: the accuracy you would get if, for every row, an oracle picked the member that got it right. If the oracle is barely above the best single model, no combiner can find much, because the correct answer simply is not present in the pool. If the oracle is far above, there is real signal to be routed and a meta-learner may capture some of it. That last check is the one people skip, and it settles the question quickly and cheaply. ## Where diversity actually comes from Ordered roughly by how much they buy you: 1. **Different inductive biases.** A regularised linear model, a nearest-neighbour model and a boosted tree ensemble carve the space in genuinely different ways: a smooth global surface, a local one, an axis-aligned piecewise-constant one. This is the biggest lever, and it is why the classic strong stack mixes families rather than tuning one family harder. 2. **Different views of the data.** Different feature subsets, different encodings of the same categorical column, different aggregation windows, different missing-value handling. 3. **Different objectives.** Optimising absolute error versus squared error, or different class weights, produces models that fail differently on the tails. 4. **Different resamples and seeds.** The cheapest and the weakest source when members are already high-capacity models fitted to convergence — enough to smooth out a little instability, not enough to change what the family cannot see. ## What a meta-learner can and cannot fix A level-1 model can reweight members, recalibrate them, and learn that one member is better in a particular region of the input space. It cannot manufacture signal that no member has. Put three near-identical members under it and the honest outcome is a small gain — sometimes a loss once the fitted weights carry fold noise. The typical symptom is a stack that lands within noise of its best member while costing three times the training and serving effort. ## The judgment this leads to When an ensemble barely beats its best member, do not add a fourth copy. Either add something structurally different and re-measure the oracle ceiling, or accept that the pool has no complementary signal and ship the single model, whose training pipeline, monitoring and latency profile are all a third the size. "Our stack is five models" is not a result. "Our stack beats the best single model by an amount that survives a hold-out the stack has never seen" is.