Random Forests

## Why the tree count behaves differently from every other knob Most model parameters trade fit against generalisation: more depth, more parameters, more iterations all eventually memorise the training set. The number of trees in a random forest is not one of those parameters, and the reason is structural. Each tree is grown independently — nothing about tree 501 depends on the errors tree 500 made. The forest's prediction is therefore the average of many draws from one random tree-building procedure, and adding draws is a **Monte Carlo** operation: it reduces the sampling noise in the estimate of that average, and nothing else. Because of this, the forest's error converges to a limit as the number of trees grows. The curve descends steeply for the first tens of trees, bends, then flattens. It does not turn back upward. This is the property Breiman established for random forests, and it is the single most-tested fact about them in interviews. ## What that looks like in practice Take a forest at 200 trees and rebuild it at 2,000 with everything else fixed. What you will see on a held-out or out-of-bag error curve is a small improvement over the first few hundred trees and then a flat line — noisy at the third decimal place, not trending. What you will also see is a tenfold increase in training time, memory footprint and per-prediction latency, because scoring a forest means walking every tree. The practical procedure follows directly: plot the error against the number of trees, find where the curve stops moving relative to its own noise, and pick a count a little past that. There is no penalty for overshooting in accuracy terms, only in cost. Many people simply pick a few hundred and move on, which is defensible; the mistake is to treat the number as a regularisation dial to be tuned down when the model overfits. ## What actually causes a forest to overfit If a forest shows a big gap between training and validation error, look at the trees, not the count: - **Depth and leaf size.** Fully grown trees drive every leaf toward purity. A forest of deep trees can still overfit noisy data, especially with few rows relative to columns. Raising the minimum samples per leaf or capping depth is the direct lever. - **The features available per split.** Allowing every feature at every split makes the trees more alike and more able to chase the same noise; restricting the subset is both a decorrelation device and, indirectly, a regularisation one. - **Leaky or target-derived features.** No ensemble parameter rescues a feature that encodes the label. The symptom looks like overfitting, but the cause is upstream in the data. A subtlety worth knowing: forests are unusually tolerant of deep trees precisely because averaging is doing the variance work. Aggressive pruning of individual trees, which is standard practice for a single tree, often costs a forest accuracy — you are attacking variance twice and paying bias for the second attack. ## Cost, and why the count still deserves thought The count is linear in three costs: training time, model size in memory, and inference latency. Training is the least worrying of the three because trees are built independently and so can be built in parallel — 500 trees across 16 cores finish in roughly a sixteenth of the sequential wall-clock time, minus overhead. Inference does not enjoy the same relief in every serving stack, and a very large forest can be slow to load and slow to score under a tight latency budget. So the honest framing is: more trees never hurts accuracy, and always costs something. Choose the smallest count on the flat part of the curve. ## Common trap in the interview Candidates often answer "no, because of the bootstrap resampling" or "no, because each tree only sees some features". Those explain why the individual trees are diverse, which lowers the limiting error, but they are not the reason the count itself is safe. The reason is that the trees are trained independently, so growing the ensemble refines an average rather than fitting the residual. Contrast that with a sequentially fitted ensemble where each new model is trained against what the previous ones got wrong: there, adding rounds genuinely can overfit, and the round count is a regularisation parameter you must tune.