What does a random forest add on top of bagged decision trees, and why does it help?

## The problem a random forest is solving A fully grown decision tree is a low-bias, high-variance model: it can carve the feature space finely enough to fit almost anything, but shift a handful of training rows and the split at the root can change, and everything below it changes with it. The classic remedy is to grow many trees on perturbed copies of the training data and average their predictions. That is bagging, and it is the base a random forest builds on. Averaging works because independent errors cancel. If you average `B` **independent** estimates each with variance `s2`, the average has variance `s2/B`, which goes to zero as `B` grows. Trees grown on resampled copies of one dataset are not independent, though — they see mostly the same rows and all of the same columns. The honest formula for identically distributed predictors with variance `s2` and **pairwise correlation** `rho` is: ``` Var(average of B trees) = rho*s2 + (1 - rho)*s2/B ``` As `B` grows, the second term vanishes and the whole expression converges to `rho*s2`. That is the ceiling on what averaging can buy you. If your trees are 90% correlated, you can add trees forever and still keep 90% of a single tree's variance. To do better you must make the trees disagree more. ## The mechanism: per-split feature subsampling The random forest change is small and precise. At **every node** of **every tree**, before evaluating candidate splits, draw a random subset of `m` out of the `p` features. The node may split only on those `m`; the other `p - m` are invisible for that decision. The next node draws a fresh subset. Consider a predictive-maintenance table where device age is by far the strongest single predictor of failure. Under plain bagging, device age wins the root split in nearly every tree, so every tree begins with the same partition and the ensemble is close to a single tree with sampling noise on top. Under per-split subsampling, device age is simply absent from a good share of nodes. Those nodes are forced to split on vibration amplitude, duty cycle, ambient temperature — secondary structure that the dominant feature was masking. The trees become genuinely different objects, `rho` falls, and the averaged prediction improves even though no single tree improved. Note the placement of the randomness: the subset is drawn **per split**, not once per tree. Drawing one subset per tree would be much cruder — a tree that never sees the informative features would be pure noise, and a tree that sees them would be an ordinary tree. Per-split drawing gives every tree access to every feature somewhere in its structure while still varying which features compete at each individual decision, which is what decorrelates the deep structure and not just the root. ## The knob and the trade The number of features considered per split, usually written `m`, interpolates between two extremes. Set `m = p` and you are back to bagged trees: maximum tree strength, maximum correlation. Set `m = 1` and each split is made on a single randomly chosen feature: minimum correlation, very weak individual trees. The conventional starting points are `m = sqrt(p)` for classification and `m = p/3` for regression, and `m` is the parameter most worth tuning. The trade is explicit. Lowering `m` **raises the bias of each tree**, because the best available feature is often not in the candidate set, and **lowers the correlation between trees**. The ensemble improves whenever the variance reduction from lower `rho` beats the bias increase. That is usually true on tabular data with many moderately informative, partly redundant features, and it is not automatic — with very few informative features among many noise features, aggressive subsampling starves most splits of signal. ## Why trees are the right base learner Variance-reduction schemes need base learners that are high-variance and low-bias, and that respond strongly to perturbation. Deep unpruned trees are the textbook case: they are unstable by construction, so there is a lot of variance to remove, and their greedy split search is easy to randomise. Averaging many high-bias, stable models (a set of near-identical linear fits, say) buys almost nothing, because their errors are the same error. ## Practical consequences Trees in a forest are grown deep and typically not pruned — pruning would attack variance a second time and buy bias you do not need, since averaging is already handling variance. Regression forests average the leaf values; classification forests normally average the per-class probabilities from each tree rather than taking a hard vote, which gives smoother scores. Because each tree is built without reference to any other tree, training is embarrassingly parallel: 500 trees across 16 cores is the same ensemble as 500 trees built one after another, just faster, and the only shared work is the data itself. That independence is a structural property of averaging ensembles, and it is what makes large forests cheap in wall-clock terms. Finally, be clear about what this does **not** fix. Feature subsampling is a variance device. If your model is underfitting because the signal genuinely is not in your features, or because the true relationship is smooth and extrapolates beyond the training range, more randomness will not rescue it — trees predict a constant outside the range they were trained on, and averaging constants gives a constant.