In gradient boosting, what do row subsampling and column subsampling per round buy you?

## The change to the algorithm Ordinary gradient boosting fits round `m`'s tree to the pseudo-residuals of **every** training row. Stochastic boosting changes one line: before fitting, draw a random fraction of the rows without replacement — say 80% — and fit the tree to the residuals of that subsample only. The resulting tree is then shrunk by the learning rate and added to the ensemble exactly as before, and it scores all rows on the next round. Column subsampling is the same idea on the other axis: each tree (or, in some implementations, each level or each split) may only consider a random subset of the features, say 60% of them. Both are cheap, both are on by default in most modern boosting practice, and both were introduced as regularisers rather than as speed tricks — though they are also speed tricks. ## What row subsampling actually buys **Decorrelation.** Successive trees in a boosted ensemble are highly dependent by construction: each one is fitted to what its predecessors left behind. If every round sees the identical data, every round chases the identical quirks in it. Resampling the rows means that a particular unlucky cluster of points only drives some of the trees, so the errors the trees make are less aligned and the summed model has lower variance. **A noisy gradient, deliberately.** The residuals of an 80% subsample are an unbiased but noisy estimate of the residuals of the whole set. Fitting to a noisy target prevents the ensemble from resolving fine detail that is not stable across resamples. This is the same reasoning that makes noisy optimisation generalise better in other settings, and it is why stochastic boosting frequently beats the full-data version on validation, not merely ties it. **Cost.** Fitting on 80% of the rows costs about 80% as much per round. On a two-million-row impression log that is a straightforward saving on every one of thousands of rounds. ## What column subsampling buys When one or two features dominate, an unconstrained split search will pick them near the root of nearly every tree, and the ensemble ends up as many near-copies of the same story. Restricting each tree to a random 60% of the columns forces some trees to find the second-best structure, which is exactly the structure that would otherwise be masked by a correlated dominant feature. The ensemble covers more of the signal, and the split search is proportionally cheaper because there are fewer candidate features to score. ## How it interacts with the other knobs - **With the learning rate.** Subsampling and a small rate are complementary regularisers, and both are usually applied together. Neither substitutes for the other: subsampling changes *what* each tree sees, shrinkage changes *how much* of each tree is admitted. - **With the stopping decision.** This is the interaction people miss. Aggressive subsampling makes the validation curve noticeably jagged, because the model is a sum of noisier increments. A jagged curve makes the round with the best held-out score less trustworthy — the apparent minimum may be a wiggle. Milder subsampling, or averaging the curve across folds, restores a clean signal. - **With rare positives.** A 30% row subsample of a dataset with a 0.5% positive rate can leave a round's tree with very few positives to learn from. When positives are scarce, subsample less aggressively. ## Choosing values There is no universal setting, but the shape of the tradeoff is predictable. Rates in the 0.5-0.9 range for rows are the normal territory; below roughly 0.5 the gradient noise usually starts costing more than the regularisation buys, and at 1.0 you have plain boosting with none of the benefit. Column fractions are chosen against how correlated and how numerous the features are: many redundant features tolerate an aggressive fraction, a handful of carefully engineered features do not. The honest way to set both is empirical: they are cheap to vary, their effect shows up clearly on a held-out curve, and unlike the round count they transfer reasonably across nearby learning rates. ## The common misconceptions Subsampling here is drawn **without** replacement per round, and its purpose is decorrelation and regularisation of a sequential fit — it is not an averaging scheme over independently trained models. It also does not guarantee that every row is used, nor that each row is used equally often; over thousands of rounds the coverage evens out statistically, but no round makes any promise about a particular row.