A readable pneumonia-risk rule list learned that asthma lowers risk — how do you respond?

## What happened A model was trained to predict mortality risk for pneumonia patients so that low-risk patients could be treated as outpatients. A readable rule-based model produced, among its rules, one saying in effect: *having asthma lowers this patient's risk of dying*. That is clinically absurd — asthma is a respiratory condition and pneumonia is a respiratory illness — and it is also, in the training data, **true**. Clinicians treated asthmatic pneumonia patients as high risk and routed them directly into intensive care. The aggressive care worked. So in the historical records, the asthma group shows *lower* observed mortality. The model was right about the data and would have been lethal in deployment: used for triage, it would have recommended sending home exactly the patients whose survival depended on being admitted. ## Name the mechanism This is not noise, not overfitting, and not a bug in the learner. It is **the outcome being contaminated by the intervention** — often called confounding by indication, or treatment leakage. The recorded outcome is not "risk of dying from pneumonia" but "risk of dying given the care this patient actually received", and the care depended on the very features the model uses. Whenever a historical label was produced under a policy that reacted to the inputs, the model learns the policy along with the phenomenon. The general test: **would this feature have influenced the treatment or the process that generated the label?** If yes, the learned relationship may be inverted relative to the causal one. Look for it in every domain where a decision intervened between the features and the outcome: past fraud reviews, past loan approvals, past manual interventions on flagged cases. ## What you actually do about it Several responses, usually in combination: 1. **Do not ship the rule.** In a rule list or a sparse model you can delete or overwrite the offending term and refit the rest around the constraint. That is a real advantage of readable models: they can be hand-edited under domain review. 2. **Constrain the direction.** If the domain says the effect of a risk indicator must not be protective, encode that as a constraint on the term's direction rather than hoping the data behaves. 3. **Fix the information set.** Restrict the model's inputs to what is known at the true decision point, before any triage response has occurred. Anything recorded after the decision, or that proxies for the decision, is leakage. 4. **Redefine the target.** Model risk under a fixed care policy, or model the untreated course of illness, rather than the observed outcome under a responsive policy. This is harder and may require a subgroup where the policy did not apply. 5. **Review every rule with a clinician, not just the suspicious one.** If one rule inverted, others did too and were merely less obviously wrong. The audit has to be systematic. 6. **Change the deployed decision.** Sometimes the honest answer is that the model may rank patients for attention but must not be permitted to *withhold* care — a decision-policy change, not a modelling change. ## Why this is an argument about interpretability, not about medicine The critical point for an interview: **a black box trained on the same records learns the same relationship**. It is in the data; a flexible model will find it. The difference is that in the ensemble it exists as a fragment of a decision surface nobody inspects, contributing quietly to millions of predictions. In the rule list it exists as a line of English that a clinician read and objected to within seconds. So the readable model did not create the problem; it *surfaced* one that would otherwise have shipped silently. That is the strongest available argument for interpretable-by-design models in high-stakes settings, and it is stronger than the usual appeal to trust or comfort. It also explains why aggregate validation metrics do not protect you here: the dangerous rule *improves* held-out accuracy, because the held-out data was generated under the same care policy. Nothing in the score curve flags it. Only a human reading the model does. ## Interview framing Resist the reflex answer of "it is spurious correlation, drop the feature". Say instead that the correlation is real, explain the mechanism that produced it, note that validation cannot catch it because the test data carries the same policy, then give the fixes and the deployment restriction. Close with the interpretability point — that the readable model is the one where the failure is visible and editable. That sequence is what a senior answer looks like here.