Why can independently perturbing correlated features produce a misleading local explanation?

## Where the perturbation goes Most local explanation methods work by asking counterfactual questions of the model: what would it have predicted if this feature had a different value, or had been replaced by a reference value? To answer, they must construct new input rows and score them. The quality of the explanation therefore depends on where those constructed rows land. Real feature vectors do not fill the input space. They concentrate on a much thinner region - the data manifold - because features are correlated. Height and weight move together; tenure and lifetime value move together; a resting heart rate and a training load move together. If a method varies one feature while holding the rest fixed, it walks straight off that region. **A concrete case.** A wearable-health risk model uses resting heart rate, average running pace, weekly active minutes and age. An explainer sweeps resting heart rate from 40 to 100 while pinning the other columns at this user's values. Somewhere in that sweep it constructs a row with a resting heart rate of 42 and a running pace of 12 km/h - a combination the training data essentially never contains. The model has no idea what to do there. It will still emit a risk score, because models always emit a score; the score is whatever the fitted function happens to do in a region no training example constrained. Tree ensembles extrapolate as flat surfaces there, linear models extrapolate straight lines, boosted models can do something jagged. None of it is knowledge. The explanation is then an accurate summary of the model's behaviour on rows that will never be scored in production. It can be internally consistent and completely uninformative about the decision you were trying to understand. ## Where the credit goes The second problem survives even if every query is on-manifold. Suppose an e-commerce churn model is given `total spend` and, separately, `orders` and `average order value`. The product of the last two is the first. The information is present twice. Whatever influence that information has on the prediction now has to be expressed across two or three columns, and how it gets divided is decided by the method's internals, not by the customer's behaviour. Change the method, change the reference values, or refit the model with the columns in a different order, and the split can move - while the model's prediction and the group's total influence are unchanged. Read the individual numbers as a ranking and you will tell a stakeholder that spend matters less than order count, which is not a finding about anything. The general statement: **local attributions are identified only up to the redundancy in the feature set.** Correlated features do not have well-defined individual credit, and no amount of extra sampling produces one, because the ambiguity is not statistical noise - it is a property of the question. ## Mitigations, and what each one costs **Sample replacement values from real data rather than independently.** Instead of sweeping a feature freely, draw substitutes from rows that resemble the target row, or condition the replacements on the features you are holding fixed. This keeps queries on-manifold. The cost is that it changes the question being answered: when you respect correlation, a feature the model never reads still receives credit because its correlated partner carries it. You have traded a nonsense-input problem for a credit-flows-to-proxies problem. Which you want depends on the purpose - reasoning about what the model computes points one way, reasoning about what a change of profile would predict points the other. Say which you chose and why. **Group redundant features and report the group.** If three columns encode one construct, report one contribution for the construct. This removes the meaningless internal split and usually makes the explanation easier for a stakeholder to act on. Grouping requires domain knowledge about which columns belong together; a correlation matrix or a clustering of the columns is a starting point, not the answer. **Prune the redundancy at the source.** Often the cleanest fix is upstream: do not carry both `total spend` and its two factors. Fewer duplicated encodings, fewer meaningless splits, and usually no loss of predictive performance. **Test robustness before presenting.** Recompute the explanation with a duplicate column removed, or with a different reference set. If the narrative flips, do not present the narrative. ## The interview answer Name both failure modes separately - impossible input rows, and undefined credit between redundant columns - because candidates who name only one usually think the fix is 'more samples'. It is not; more samples make an off-manifold estimate more precise, and precision on the wrong question is still the wrong answer.