Trees, Forests, and Boosting

Out-of-bag error is trustworthy when rows are independent and the ensemble is large, because each row is scored only by base models that never saw it. It misleads when rows are grouped, duplicated or time-ordered.

Both permit roughly 64 leaves, but they constrain different things. A depth cap of 6 limits every prediction path to six splits. A cap of 63 leaves permits a chain 62 splits deep.

Leak-free encodings fix the features, not the gradients. Standard boosting computes a row's residual from an ensemble already fitted on that row, so residuals are optimistic. Ordered boosting scores each row with a model trained only on the rows preceding it.

Size leaves by expected positive cases, not total rows: at a 4% rate, 50-row leaves hold about two positives, so their probabilities are noise. Divide the events you need behind a prediction by the prevalence.

With 400 features, sqrt(p) offers only about 20 candidates per split. If about ten probes carry the signal, most splits see none and are made on noise — the trees end up diverse but uninformative. Raise the features per split.

A split happens only if its gain - the drop it produces in the regularized objective - clears the per-leaf cost and the minimum-gain threshold. A large L2 penalty on leaf scores, a high threshold, or a child-curvature floor each veto splits early.

Treat them separately. The stopping fold chose a hyperparameter, so its score is optimistic and the reported accuracy must come from data that played no part in stopping. Ship 740 as a fixed round count, valid only for the learning rate and subsampling it was found under.

Because their errors are almost the same errors. Members with the same inductive bias, features and objective get the same rows wrong, and combining only cancels mistakes that members make independently. Diversity is the lever, not the number of models.

Choose on constraints, not on accuracy claims. LightGBM for training throughput on very large, wide tables; CatBoost for high-cardinality categorical columns and fast uniform-depth inference; XGBoost as a strongly regularized general default. Tuned properly, their accuracies usually land within noise.

Weigh the bias it removes against wall-clock and memory. The prediction shift shrinks as rows accumulate, so on tens of millions of rows it buys little; on small, categorical-heavy data with many rare levels it can move the holdout number.

Averaging predicted probabilities beats majority voting when the base models differ in confidence: a few strongly negative models can outweigh a bare majority of barely-positive ones. Voting collapses each model to one ballot and discards that information.

An oblivious tree uses the same feature and threshold at every node of a level, so a depth-d tree holds d tests and 2^d leaves. Scoring becomes d comparisons plus one array lookup, and the constrained shape acts as regularisation.

Pre-pruning judges each split on its immediate gain with no lookahead, so a near-worthless split that unlocks a strong one a level below is refused. Post-pruning grows first and judges the pair together.

Extremely randomized trees draw each candidate feature's cut-point at random instead of searching for the best threshold, and usually train on the full sample. That adds bias per tree, removes more variance, and makes training much faster.

Two tree-native mechanisms exist. CART learns surrogate splits, backup features whose cut best reproduces the primary split, and uses the best available one. Boosted trees instead learn a default direction per node by testing which side gives more gain.

The tree only chooses which rows move together; the leaf value is the constant that actually minimises the chosen loss for those rows given the current model. Under squared error that is their mean, under absolute error their median.

A histogram-based tree can only cut at bin edges. With 255 equal-count bins the top bin already holds about 0.4 percent of rows, so a real threshold at the 99.9th percentile falls inside it.

Boosting fits whatever the ensemble still gets wrong, and a mislabelled row is permanently wrong, so later trees carve tiny regions around those rows. Training loss collapses because the noise is being memorised; validation loss turns up because those rounds add nothing real.

A tree's splits are each chosen to minimise impurity at that node alone, with no lookahead, so the locally best cut can foreclose a better pair of cuts beneath it. Finding the globally optimal tree is NP-hard.

Only a narrow one: for this fitted model, on its training rows, splits on that column removed the largest share of impurity, given the other columns present. It is not a causal effect, not signed, not stable, and not a property of the data.

Constrain only where the direction is a domain law or a written underwriting policy, such as risk never falling as debt-to-income rises. Each constraint buys behaviour you can promise and defend, and usually costs a little held-out accuracy.

Price both sides. Measure the stack's p99 latency, including fan-out and feature lookups, against the share of the 40 ms the model owns, and measure its gain in the business metric on untouched data. Ship the smallest ensemble clearing both.