When would you choose a random forest over gradient boosted trees on a tabular dataset?

## Two ensembles, two different jobs A random forest and a gradient boosted tree ensemble are both collections of decision trees, but they combine trees for opposite reasons, and that difference drives every part of the choice. A **random forest** grows many trees independently. Each tree is fitted on a bootstrap resample of the rows (sampling with replacement, so roughly 63% of the distinct rows appear in any one tree), and at every split it may only consider a random subset of the columns. Each individual tree is grown deep: low bias, high variance. Averaging their predictions (or majority-voting them) cancels much of that variance, because the trees' errors are only partly correlated. **Gradient boosting** grows trees one at a time. Each new tree is fitted to the negative gradient of the loss with respect to the current ensemble's predictions — the pseudo-residuals — and its contribution is shrunk by a learning rate before being added. Individual trees are deliberately weak (shallow); the additive sum of many of them drives down **bias**. Conceptually: `F_new = F_old + lr * tree(negative_gradient)`. ## Axis 1 — tuning effort A forest has few knobs that matter: the number of trees (more is never worse for accuracy, only slower), how many features are considered per split, and a minimum leaf size. Out-of-the-box settings usually land within a small margin of what tuning would buy you. A booster's knobs interact. Learning rate trades off against number of rounds; depth or leaf count controls how much interaction each tree can express; row and column subsampling and explicit L1/L2 penalties on leaf weights control overfitting. An untuned booster can sit far from its own best. Concretely: a 50,000-row, 80-column commercial-property pricing table with **two engineer-days**. A forest gets you a trustworthy model and an out-of-bag error estimate in an hour. A boosted model with early stopping and a modest search will usually beat it on error — but most of those two days go into search plumbing and validation discipline, and if the deadline slips you have nothing to ship. ## Axis 2 — noise tolerance This is the axis candidates most often miss. Boosting fits what the ensemble is still getting wrong. A mislabelled or freak row keeps producing a large gradient, so successive trees keep spending capacity on it — boosting will happily memorise noise. A forest sees that row in only some of its bootstrap samples, and averaging dilutes its influence. On a **noisy 1,500-row survey extract** — self-reported answers, a small sample, real label noise — an untuned booster is routinely beaten by a plain forest. Boosting can be brought back with a low learning rate, shallow trees, aggressive subsampling and early stopping on a held-out fold, but the forest's advantage there is essentially free. ## Axis 3 — parallelism and training cost Forest trees are independent, so training is embarrassingly parallel across cores or machines, and you can stop early or extend the ensemble later without retraining. Boosting rounds are inherently sequential — round t needs round t-1's predictions — so parallelism lives *inside* a round, in the split search across features. On a fixed core budget, a forest usually gets to "good enough" in less wall-clock time. Inference cost differs too, but not in one predictable direction: forests use many deep trees, boosters typically use shallow trees but more of them. Measure per-row latency at the ensemble size you would actually ship. ## Overfitting behaviour — the tested fact Adding trees to a **random forest** does not overfit: test error decreases and then flattens as the average converges. Adding rounds to a **booster** does overfit: test error falls, bottoms out, and then rises. That is why every serious boosted model is trained with early stopping against a validation fold and why "how many trees?" is a free choice for one method and a tuned choice for the other. ## What does not decide it Both are non-parametric, both are invariant to monotone rescaling of features so neither needs normalisation, and both capture feature interactions without you specifying them. Both expose feature importances of comparable quality. So "handles mixed scales" or "captures non-linearity" is not an argument for either one over the other. ## A defensible default Fit the forest first: it is the baseline that is hard to get wrong, and its out-of-bag estimate costs nothing. Then attempt a boosted model with early stopping and a bounded search. Ship the booster only when it beats the forest on the *same* cross-validation protocol by a margin you would defend to a stakeholder — and remember that on small or noisy data, the honest result is often that it does not.