Why does mean impurity decrease rank a pure-noise, near-unique donor ID column third?

## The two ingredients of the bias **1. More candidate cut-points means more chances to win.** A greedy splitter evaluates every admissible threshold for every feature and keeps the best-scoring one. A binary flag offers one candidate split. A continuous or near-unique column with 5,000 distinct values offers on the order of 5,000. Even when the column is independent of the target, the *maximum* impurity drop over thousands of random candidates is comfortably positive - this is a selection-of-maxima effect, not a property of the column. The splitter is comparing one honest candidate against thousands of lottery tickets. **2. The score is a training-fit statistic.** Nothing in mean decrease in impurity (MDI) is validated. A split that isolates eleven donors who happen to have given twice reduces training Gini exactly as legitimately as a split on genuine signal, and it collects credit. On a 5,000-row table a near-unique identifier can eventually isolate almost anything. Put together: the noise ID wins many splits it has no business winning, and every one of them pays. ## Why node weighting does not save you Each split's contribution is scaled by `n_node / N`, the share of training rows reaching the node, so the deep splits where the ID does its damage are individually tiny. The catch is volume. A fully grown tree on 5,000 rows contains hundreds of nodes, the great majority of them deep and small, and the ID is a plausible splitter at nearly all of them, whereas a genuinely decisive feature is consumed near the root and has little left to contribute below. Hundreds of small positive contributions add up to a respectable-looking total. This is also why a genuinely decisive binary flag can be *beaten* by a mediocre continuous feature with 2,000 cut-points. The flag delivers one large drop high in the tree and is then exhausted - below that split, its value is constant within each branch, so it can never be chosen again on that path. The continuous feature is reusable at every node with a fresh threshold, and its many small drops out-total the flag's single big one. MDI measures accumulated split usefulness, not decisiveness. ## The depth-4 contrast as a diagnostic Refit the same data with depth capped at four and re-read the ranking. A depth-4 tree can afford at most fifteen internal splits, and the greedy rule spends them on the highest-gain candidates available at large nodes - which is where real signal lives. A noise ID that ranked third in the fully grown forest usually falls sharply, while features carrying genuine signal keep their positions. That contrast is the tell: **a feature whose importance depends on the trees being allowed to grow deep is a feature earning its score from memorisation.** Two honest caveats. The depth-capped model is a different model, so you are comparing rankings, not proving anything about the original. And capping does not drive the bias to zero - a high-cardinality column can still win a shallow split by luck, particularly on small data. ## The rest of the diagnostic kit - **Profile cardinality alongside the ranking.** Print distinct-value counts next to the top ten scores. When the top of the list correlates with cardinality, treat the whole ranking as suspect. - **Check stability.** Refit with several seeds or across cross-validation folds. Genuine drivers hold their positions; artefacts wander. - **Sanity-check against the validation metric.** If the model's held-out performance is unchanged by whether the ID column is present, the column is contributing nothing real however high it ranks. - **Watch for leakage-shaped columns.** Row identifiers, timestamps stored as integers, and case numbers often correlate with the target through collection order, which produces the same symptom with a different cause and deserves a different fix. ## What to do about it The cheapest structural fix is not to feed near-unique identifiers to the model at all - an ID is not a feature, and its high rank is a warning that it was allowed in. Where a high-cardinality column is genuinely meaningful, constrain the splitter: cap depth, raise the minimum samples per leaf so tiny noise-carving nodes cannot form, and bin the values so the number of candidate cut-points shrinks. Each of these reduces the number of lottery tickets rather than trying to correct the score after the fact. What you must not do is take the ranking at face value and narrate a story about donor identifiers mattering. The number is real arithmetic over a real fitted model; it just does not mean what the reader assumes.