Your model card lists impurity importance as the key drivers - what claims does that ranking support?

## The claim the arithmetic actually licenses An impurity importance score is a sum of weighted impurity drops over the splits a fitted model made on one column, computed on the rows it was trained on. Unpacked, a published ranking supports exactly this sentence: > Within this fitted model, trained on this snapshot with this configuration, splits on column X accounted for the largest share of the training impurity the model removed, given that all the other columns were also available. Every qualifier in that sentence is load-bearing, and each one blocks a claim readers routinely make: - **"Within this fitted model"** blocks "X is the most important predictor of the outcome". Refit with a depth cap, a different seed or a different algorithm and the ordering can change. - **"On this snapshot"** blocks any claim of durability. Importance drifts as the population drifts. - **"Share of impurity removed"** blocks percentage-of-outcome readings. It is not variance explained and not a contribution to accuracy. - **"Given the other columns"** blocks "X matters, Y does not". Two near-duplicates split their credit and both look mediocre; drop one and the survivor jumps. Low rank can mean redundant, not irrelevant. - Nothing in the sentence is signed, so it never supports "more X means more outcome". - Nothing in it is causal, so it never supports "raise X and the outcome follows". ## What to require before publishing **Pin the definition.** Summed gain, split counts and coverage-weighted measures rank the same model differently. A card that says "feature importance" without naming which one, how it was normalised, which data snapshot and which seed, has published an unreproducible number. Choose one definition as the house standard so rankings are comparable across the team's models, and record the rest. **Publish stability, not just order.** Refit across seeds and cross-validation folds and report the range of each feature's rank. In most tabular models the top two or three are stable and the middle is noise. A card that shows rank intervals is honest; a card that shows a clean ordered list of twenty features implies a precision that does not exist. **Audit the top of the list before you defend it.** Print distinct-value counts alongside the scores - if rank tracks cardinality, the ranking is measuring the splitter's opportunity set. Look for near-duplicate columns diluting each other. Check whether the ordering survives a depth-capped refit; a feature that only ranks high when trees grow deep is earning its place from memorisation. **State the training-fit basis in the card text itself.** Not in an appendix. The single most common misreading is that these numbers were validated, and one sentence prevents it. ## The organisational call The hard part is rarely the statistics; it is that a ranking is a compelling artefact. It is short, ordered and numeric, so it travels into decks and gets read as a list of causes. Three decisions worth owning as a lead: **Decide what the ranking is allowed to be used for.** Legitimate uses are model debugging, sanity-checking against domain expectation, monitoring for shifts between refits, and documentation. Deciding to intervene on a feature is not among them, and neither is telling a business that a column "drives" an outcome. **Give the escalation a standard answer.** When someone asks to act on the number three feature, the answer is not "the model says so" and not a flat refusal. It is: the model learned an association under the regime that generated this data; whether changing the feature changes the outcome is a different question that needs designed evidence - an experiment, a natural experiment, or an explicit causal argument with its assumptions stated. Offer to help scope that instead. **Accept a boring card over a persuasive one.** Rank intervals, a named definition and a stated in-sample basis make a duller artefact than a clean top-ten bar chart, and they are the reason the card survives contact with a reviewer, an auditor or a model risk function. ## The failure mode to name out loud The worst outcome is not a wrong ranking; it is a right ranking that nobody can reproduce and everybody over-reads. A number computed correctly, published without its qualifiers, and repeated until it becomes an organisational belief is harder to unwind than an obvious error - because there is no moment where it visibly breaks.