Why do boosted tree ensembles lead on heterogeneous tabular data?

## What "heterogeneous tabular data" actually means A typical business table has columns like `price_per_square_metre`, `days_since_last_contact`, `region_code`, `is_renewal`, `contract_type`. They have different units, different scales, different types, and no relationship to one another beyond belonging to the same row. Column 7 and column 8 are not neighbours in any meaningful sense; shuffling the column order changes nothing about the problem. That is the opposite of an image or a waveform, where adjacency carries meaning and the signal is locally smooth. Almost everything that makes boosted trees the default on tables follows from this one structural fact. ## Property 1 — a split is a threshold, not a distance A decision tree node asks a single question: `is feature_j <= t?`. That comparison depends only on the *order* of the values in that column, so any monotone transform — changing dollars to thousands of dollars, taking a logarithm, applying a rank transform — yields exactly the same partition. Consequences: - No standardisation or scaling step, so no scaler to fit, persist and keep in sync at serving time. - Skewed and heavy-tailed columns need no transformation. - Outliers in the *feature* have limited effect: a value ten times too large is simply on the far side of a threshold, not a coordinate that drags a distance calculation or a fitted coefficient. Methods built on distances or on weighted sums of features have none of this for free; they need every column brought onto a comparable scale before they behave. ## Property 2 — real tabular relationships are often thresholded Business processes create edges: an approval rule at a credit score, a discount above a quantity, a fee that changes at a tenure boundary, a policy that changed on a date. A piecewise-constant model made of thresholds represents that natively. A model that assumes a smooth or linear relationship must approximate a step with a curve, and typically smears it. The flip side is the honest weakness: a genuinely smooth or linear trend has to be approximated by a staircase of splits, and a tree can never extrapolate beyond the range of the training data — outside it, every prediction is the value of the nearest leaf, flat forever. ## Property 3 — interactions come for free Splitting on region and then, within that branch, on contract type expresses "the effect of contract type depends on region" without anyone specifying the interaction. On tables where the signal lives in a handful of conditional effects, this is worth more than any amount of coefficient tuning in an additive model. Depth controls how many features an interaction may involve. ## Property 4 — mixed types and missing values pass through Categorical codes, ordinal levels, binary flags and continuous measurements coexist in one model. Missing values can be handled by sending them down a learned default branch at each split rather than being imputed with a guess that invents information. In real tables missingness is often informative — the field is empty *because* of something — and a split-based model can exploit that directly. ## Property 5 — boosting spends capacity where the error is Bagging averages equally capable trees. Boosting fits each new shallow tree to the negative gradient of the loss at the current predictions — in plain terms, to what the ensemble is still getting wrong — and adds it scaled by a small learning rate. Capacity therefore accumulates in the regions that remain hard, and the shrinkage makes the whole thing a slow, controllable descent rather than a leap. Combined with explicit penalties on leaf weights and row/column subsampling, you get a high-capacity model with genuine control over how much of that capacity gets used. ## Why the same table is unremarkable for other families Families that lean on smoothness, on locality (nearby points behave alike), or on a linear-additive functional form are all making an assumption the data does not satisfy: tabular feature space has no geometry to be smooth *in*, distances across mixed units are close to meaningless without careful scaling, and the true function has edges. Those families are not weak in general — they are strong where their assumption holds. On heterogeneous tables it does not, and the assumption that does hold — "the target is a sum of thresholded, interacting effects" — is exactly the one a boosted tree ensemble encodes. ## Where the argument stops This is an argument about typical tables, not a law. Very high-dimensional sparse features, strictly monotone requirements imposed by regulation, needing to extrapolate a trend beyond the observed range, or a hard interpretability constraint all pull elsewhere — and on very small, very noisy tables the boosted model may not even beat a simpler ensemble. Say that out loud: an interviewer is listening for whether you know the boundary of your own default.