An attribution map barely changes after a trained network's top layers are randomly reinitialised — what does that mean?

## The problem the check solves Attribution maps are evaluated by looking at them, and the thing a human finds convincing is a map that traces the object. But tracing the object is also what a plain edge detector does, and an edge detector explains nothing about a model. So visual plausibility cannot distinguish a faithful explanation from an input-processing artefact. A sanity check is a *falsification test*: a condition an explanation method must pass to be worth interpreting at all, not a measure of how good it is. ## The model-parameter randomization test The logic is a controlled ablation on the model rather than on the input. 1. Compute the attribution map for a fixed input and class on the trained network. 2. Replace the learned weights of the topmost layer with fresh random values, leaving everything else trained. Recompute the map. 3. Continue downward, randomising successively more layers (the *cascading* variant), or randomise one layer at a time with the rest trained (the *independent* variant), recomputing the map at each stage. 4. Measure the similarity between each map and the original with a quantitative metric — rank correlation over pixel scores, structural similarity, or the overlap of the top-k attributed pixels. After randomising the top layers, the network's output for that class is arbitrary. An explanation of an arbitrary output must itself be arbitrary. A method whose maps stay highly correlated with the original has failed: it is a function primarily of the input, with the model's parameters entering only weakly. ## The data randomization test The complementary check attacks the labels instead. Train the same architecture on a copy of the dataset with the labels randomly permuted. Such a model can only memorise; it has learned no real input-label relationship. A faithful attribution method should produce maps on this model that look nothing like the maps from the properly trained model. If they match, the method is again reading the input, not the model. ## What tends to fail, and why Methods that modify the backward pass to suppress negative signals produce beautifully sharp, object-shaped pictures — and that sharpness is largely a property of how the network's early layers process edges, not of what the classifier learned. Empirically, plain input gradients and class-weighted feature-map methods such as Grad-CAM change substantially under parameter randomization, while guided-backpropagation-style maps and their products with a coarse class map are far less sensitive. The lesson generalises past the specific list: **the more a method's visual appeal comes from post-processing the backward signal, the more suspicious you should be**, and any new method should be put through the same test before it is adopted. ## Reading the result correctly - **Passing is necessary, not sufficient.** A map that changes when the weights change is at least a function of the model. That does not make it a correct account of the decision, and it does not license any causal claim. - **Failing does not make the picture useless — it makes it not an explanation.** A method that behaves like an edge detector may still be a fine visualisation of image structure. It simply must not be captioned "this is what the model looked at", shown to a reviewer, or used to justify a deployment decision. - **Judge numerically.** Two heatmaps can look different to the eye while being highly rank-correlated, and vice versa; the normalisation and colour map you use to display them can manufacture apparent differences. Report a similarity number per randomisation stage. - **Check the class you care about.** After randomisation the argmax class changes, so keep explaining the *originally predicted* class index throughout; otherwise you are comparing explanations of two different questions and the test is meaningless. ## Where this sits in a workflow Run the check once per attribution method per architecture family, not once per image — it is a property of the method-and-model pairing, and it is cheap relative to the cost of building a review process on top of an explanation that turns out to be an edge detector. Combine it with two cheap habits: cross-method agreement (do two independent methods point at the same region?) and intervention (mask the highlighted region and measure how far the class probability actually drops). A map that survives randomisation, agrees with a second method, and predicts a measurable drop under masking is worth acting on. One that only looks convincing is not.