How do you rank adverse-action reason codes when correlated features split the attribution?

## The symptom A lender's decline explanations are generated by attributing each decision to its inputs and reporting the top four. In testing, two things show up. First, an application declined mainly because of credit-line usage gets no usage reason at all — the top four are dominated by unrelated inputs. Second, rescoring the same applications after a small model refresh permutes the reported reasons, even though the approve/decline outcome for nearly every application is unchanged. Both are the same underlying phenomenon: correlated inputs share credit for the same underlying story, and a per-input ranking treats them as competitors. ## Why the split happens A credit file naturally carries several near-redundant views of one fact. Current revolving balance, utilization as a percentage of limit, and available headroom in dollars are three encodings of the same behaviour. Any attribution method that assigns credit per input must divide the effect between them somehow; none of them can receive the full effect, because holding the other two fixed leaves little for the third to explain. So the family carries, say, 30% of the push toward decline, but appears in the ranking as three entries of about 10% each — below a single delinquency input that carries 18% on its own. The list is arithmetically defensible and communicatively wrong: the biggest thing about this applicant does not appear in the explanation they receive. The churn follows from the same structure. When three quantities sit within noise of one another, an insignificant change — a retrain, a slightly different sampling of the training data, a rounding difference — reorders them. Ranking is discontinuous where the underlying quantities are nearly equal. ## Why churn matters even though the decision does not change It is tempting to shrug: the applicant was declined either way. Three reasons not to. - **Reproducibility.** If a notice is challenged, you must be able to show why those four reasons and not four others. `The ordering is unstable` is not a defence. - **Consistency across applicants.** Two applicants with materially identical files, scored a week apart, should not receive different explanations. That is the kind of inconsistency an examiner probes. - **Trust and remediation.** An applicant told to fix the wrong thing does the wrong thing and reapplies to the same decline. ## The remedy: rank codes, not inputs The durable fix is a reason-code taxonomy defined before modelling and owned jointly by the business and compliance. 1. **Define the codes** as customer-meaningful families — credit-line usage, recent delinquency, length of credit history, recent inquiries, income relative to obligations — each with fixed, reviewed wording. 2. **Map every input to exactly one code.** The mapping is a reviewed artefact, versioned with the model. New input, new mapping entry, or it does not ship. 3. **Aggregate then rank.** Sum the per-input contributions within each code, and rank the codes. The usage family now carries its full 30% and appears first, which is what the applicant actually needs to hear. 4. **Make the ranking deterministic.** Round contributions to a stated tolerance and break remaining ties by a fixed documented code priority, so the same application always yields the same list. On points-based scorecards the classical version of this is ranking by points below the maximum achievable for each characteristic — how much the applicant lost relative to the best attainable value — with the same aggregation over related characteristics. The principle carries over unchanged to a more flexible model: rank by shortfall attributable to a code, not by an input's raw value. ## Testing stability before release Treat explanation stability as a release check with a number attached, alongside the usual predictive metrics. - Rescore a frozen sample of past declines under the candidate model and the incumbent. - Measure the fraction whose **top code changes**, and the fraction whose **top-four set** changes (order within the set matters less than membership). - Investigate any code that fires on nearly every decline: a code that appears on 90% of notices is too coarse to be informative and probably needs splitting. - Investigate codes that never fire: dead entries in the taxonomy usually mean a mapping mistake. Set a tolerance, and make a large jump in top-code churn something a human reviews before release rather than something applicants discover. ## What not to do - **Do not drop correlated inputs from the model** to tidy the explanation. That trades predictive quality for reporting convenience, and the surviving input's contribution becomes a proxy for the family anyway. - **Do not rank by raw feature value.** A large balance is not the same as a balance that drove this decision; that is a magnitude, not a contribution. - **Do not let the order be whatever the scoring code returns.** Implicit ordering is unreviewable and changes with unrelated refactors. - **Do not report every input with a non-zero contribution.** A long list is legally weaker and practically useless. ## What the interviewer is checking That you notice the difference between the decision being stable and the explanation being stable, that you locate the fix in the reason-code taxonomy rather than in the attribution maths, and that you treat explanation stability as something you measure and gate on rather than something you hope for.