What does it mean for a local explanation of one prediction to be faithful rather than plausible?

## Two different questions When you produce an explanation for a single prediction - a list of features with signed contributions, a highlighted region of an image, a small rule - there are two separate questions you can ask about it, and interviews on this topic exist mainly to check that you keep them apart. **Faithfulness** asks: does this explanation describe what the model actually did? The reference is the model. A faithful explanation of a single row identifies the inputs the model's output genuinely depended on at that point, in roughly the proportions it depended on them. **Plausibility** asks: does this explanation look right to a human? The reference is the reader's domain knowledge. A plausible explanation names features a domain expert would have named. These are independent properties. Every combination exists: - Faithful and plausible - the happy case, and the one people assume they are in. - Faithful but implausible - the model really is leaning on a feature nobody expected. This is uncomfortable but it is *information*, and it is often how leakage and shortcut features get found. - Plausible but unfaithful - the dangerous case. The explanation names exactly the features the reviewer already believed in, so the review stops. - Neither - usually spotted quickly, because implausible explanations get scrutinised. ## Why the dangerous case is dangerous A plausible explanation is *self-ratifying*. The reviewer's own prior is the only check being applied, so an explanation that matches the prior passes review by construction. That is the mechanism behind the husky-versus-wolf demonstration from the LIME paper: a classifier that seems to distinguish the two animals is actually responding to snow in the photo background, because in the training images wolves were photographed on snow. An explanation highlighting fur texture or muzzle shape would be received as obviously correct. The model would still be broken. The lesson is not about images. Any post-hoc explanation of any model can be plausible without being faithful, and the more domain-shaped the feature names are, the more easily a wrong explanation slips through. A churn model that reports 'days since last login' as the driver will be believed whether or not that is what the model used. ## Testing faithfulness instead of assuming it Because plausibility is free and faithfulness is not, faithfulness has to be measured. Two families of check are standard: **Perturbation or deletion checks.** Take the features the explanation ranks highest, degrade them - replace with a baseline value, mask the region, corrupt them - and re-score the row. If the prediction moves substantially more than when you degrade the same number of randomly chosen features, the ranking is carrying real signal. If the two curves overlap, the explanation is not tracking the model. Caveat, and a good one to raise unprompted: deleting or replacing values itself creates input rows the model never saw during training, so the test has some of the same off-distribution weakness as the explanation methods it is auditing. Averaging over many rows and comparing against the random baseline is what makes it usable. **Model-randomisation sanity checks.** Recompute the explanation after scrambling the trained model's parameters. If the explanation looks essentially the same for the trained model and a randomised one, it is a function of the input, not of the model, and it cannot be faithful to anything. Published work on saliency maps found several popular methods failing exactly this check. A third, cheaper option in low-stakes settings: fit a model that is transparent by construction on the same data, and see whether the post-hoc explanation of the complex model at least agrees with it in the region you care about. Disagreement does not prove unfaithfulness, but it is a prompt to look harder. ## The boundary worth stating out loud Even a perfectly faithful explanation tells you about the model, not about the world. 'The model's output depended heavily on this feature for this row' is not 'changing this feature would change the outcome'. The model may be leaning on a proxy that is only associated with the outcome in the training data. Turning attributions into claims about real-world effect requires causal analysis, which is separate machinery entirely; the correct interview move is to name that boundary and stop, not to blur it. ## What to say in the room Define both terms, say they are independent, give the plausible-but-unfaithful case as the one that costs money, and name at least one concrete test you would run before an explanation is trusted. Candidates who say 'the explanation looked sensible so the model is fine' have answered the wrong question.