How is a random forest's built-in impurity importance computed for one feature?
answer
- credit accrues split by split
- parent impurity minus weighted children
- scale by rows reaching the node
- summed over trees, then normalised
- training rows only, never held out
basics
~20 sImpurity importance sums, for every split that uses the feature, the drop in node impurity weighted by the fraction of training rows reaching that node. The totals are combined across the forest's trees and normalised to sum to one.
solid answer
~50 sEvery time a tree splits on a feature it records a weighted impurity decrease: `(n_node / N) * (impurity(node) - (n_left/n_node)*impurity(left) - (n_right/n_node)*impurity(right))`, where impurity is Gini or entropy for classification and within-node variance for regression. Mean decrease in impurity (MDI) sums that quantity over every split on the feature in a tree, then combines it across all trees, and implementations normally normalise the vector so the scores sum to one. Two consequences follow directly from the definition. It is computed entirely on the rows the trees were fitted on, so a split that only fits noise still earns credit; and it is unsigned and relative, so a score of 0.30 means the feature accounted for roughly 30 percent of the total weighted impurity the forest removed, not that it raises the prediction or that it explains 30 percent of anything out of sample.
code
python · 14 linesdef gini(counts):
n = sum(counts)
return 1.0 - sum((c / n) ** 2 for c in counts)
parent = [40, 60] # 100 rows reaching this node
left, right = [30, 10], [10, 50] # after one split
n = sum(parent)
drop = (gini(parent)
- (sum(left) / n) * gini(left)
- (sum(right) / n) * gini(right))
weighted = (n / 1000) * drop # 1000 rows grew the tree
print(round(gini(parent), 4), round(drop, 4), round(weighted, 4))go deeper
Be ready to say what the number is in one sentence: the share of training impurity reduction credited to splits on that column. Know that it is relative, unsigned, and computed on the training data.
You are expected to write the per-split formula, name Gini, entropy and variance as the impurity functions, and explain why each split is weighted by the fraction of rows reaching its node before the contributions are summed and normalised.
Show that you read a ranking with its fitting configuration in hand: depth caps, leaf minimums, tree count and seed all change which splits exist, so you quote scores alongside how stable they were across refits rather than as fixed facts.
Own the reproducibility angle. Impurity importance has several definitions that rank differently, so a team that publishes 'feature importance' without naming the definition, the normalisation and the data snapshot has published something nobody can reproduce or contest.
## The quantity being accumulated A decision tree grows by repeatedly choosing a feature and a threshold that make the two child nodes purer than the parent. "Purity" is measured by an impurity function: - **Gini** for classification: `Gini = 1 - sum(p_i^2)` over the class proportions `p_i` in the node. It is 0 when the node holds a single class and 0.5 for a balanced binary node. - **Entropy** for classification: `H = -sum(p_i * log2(p_i))`, 0 for a pure node and 1 bit for a balanced binary node. - **Variance / squared error** for regression: the mean squared deviation of the target around the node mean. For one split, the impurity decrease is the parent's impurity minus the sample-weighted average of the two children's impurity: `decrease = impurity(parent) - (n_left/n_node)*impurity(left) - (n_right/n_node)*impurity(right)` That number is then scaled by how much of the data the node holds, `n_node / N`, where `N` is the number of rows used to grow the tree. A split near the root that separates thousands of rows therefore counts far more than a split at depth twelve that separates nine rows. ## From one split to a feature score Mean decrease in impurity (MDI) for feature `f` in one tree is the sum of those weighted decreases over every node in the tree that split on `f`. A feature used at four nodes collects four contributions; a feature the tree never chose collects nothing and scores exactly zero. For a forest the per-tree scores are combined across the ensemble - implementations typically normalise each tree's vector, average over trees, and expose a vector that sums to one. The published number for a feature is therefore best read as **the share of the total weighted training impurity the ensemble removed that is attributable to splits on this column**. The same accounting appears in gradient-boosted models under the name **total gain**, with the split's loss reduction standing in for the impurity drop. The mechanics - accumulate a per-split quantity, attribute it to the split's feature, sum - are identical. ## What the definition immediately implies **It is a training-fit statistic.** Nothing in the formula ever looks at held-out data. If a tree grows deep enough to carve noise into pure leaves, every one of those noise splits contributes a positive decrease. MDI cannot distinguish signal from memorisation, because both reduce training impurity. **It is unsigned.** MDI says the column was useful for separating the target; it never says higher values push the prediction up. A feature with a strong U-shaped effect and a feature with a monotone effect can score identically. **It is relative and conditional on the fitted model.** Because scores are normalised, adding a strong new column pushes every existing score down without anything changing about those columns. And a score reflects the value of the feature *given the other columns that were available* - a feature that is perfectly predictive but shadowed by a near-duplicate can score low. **Zero means "never split on", not "irrelevant".** That is the single most common misreading. If two columns carry the same information, a tree that always picks the first leaves the second at zero. **It depends on how the trees were grown.** Depth caps, minimum leaf sizes and the number of trees all change the set of splits that exist, and therefore change the ranking. Two forests fitted with different seeds on the same data will not produce the same ordering, especially in the middle of the ranking where score differences are small. ## Reading a score in practice A sensible reading of "utilisation ratio: 0.22" is: across the trees in this forest, roughly 22 percent of the weighted training impurity that got removed was removed by splits on utilisation ratio, given the other features present, under this fitting configuration. Everything a stakeholder usually wants - does it matter out of sample, in which direction, would removing it hurt - is a different question that this arithmetic does not answer. ## Common variants you may be shown Besides the summed decrease, implementations expose a **split count** (how many nodes used the feature at all, ignoring how much each bought) and coverage-style measures weighting each split by how many rows it touched. These rank features differently from total gain, so a ranking is only reproducible if you state which definition produced it.
- Which impurity function is used when the trees are regression trees?Within-node variance of the target, equivalently the mean squared error around the node mean. The split's decrease is the parent's variance minus the sample-weighted variance of the two children, and the accumulation over splits and trees is exactly the same as in the classification case.
- Why can the ranking change when you refit the same forest with a different random seed?Splitting is greedy and near-ties are broken arbitrarily, and each tree sees a different bootstrap sample. When two columns are close in quality, which one wins a node flips between refits, so their scores swap. Treat the middle of a ranking as noise unless you have checked it across several refits.
- A feature scores exactly zero. What can you conclude?Only that no tree in the forest ever split on it. That happens when the column is genuinely uninformative, but also when a near-duplicate column always won first, or when the column's values are too coarse to beat competitors at any node. Zero is not evidence of independence from the target.
saying these in an interview costs you the question
- Claims the score shows the direction of the feature's effect
- Thinks impurity importance is measured on held-out data
- Says a zero score proves the feature is unrelated to the target
- Reads the normalised score as a percentage of model accuracy
- Confuses the number of splits with the total impurity removed