A pneumonia classifier's Grad-CAM maps highlight a scanner watermark, yet held-out accuracy is high — do you ship it?

## What the map is actually telling you A class-discriminative heatmap concentrated on a corner artefact — a burned-in scanner marker, an equipment watermark, a positioning label — is one of the highest-signal findings an attribution map can produce, because the region is *known a priori to carry no diagnostic information*. Unlike a map over lung tissue, whose plausibility you cannot judge, this one is falsifiable by inspection: nothing about a manufacturer's mark can be evidence of disease. The mechanism is almost always dataset provenance. Sicker patients are imaged with portable equipment at the bedside; cases from an outbreak period arrive from one referring site; a positive-heavy cohort was assembled from a specialist centre. The artefact is then correlated with the label, it is trivially easy to detect compared with subtle parenchymal texture, and gradient descent finds it first. The model has not cheated — it has optimised exactly what you asked it to. ## Step one: is the map even trustworthy? Before acting, verify the attribution method on this model: confirm the map changes when the trained weights are randomised, and check that a second, independent attribution method points at the same region. Acting on a map that turns out to be an input artefact wastes an investigation. One image is also an anecdote; compute maps across a sample of positives and quantify how often the artefact region carries the top attribution mass. ## Step two: convert the picture into a measurement Attribution maps do not support causal claims. Interventions do. - **Ablate the region.** Mask, crop or inpaint the artefact and re-evaluate on the same held-out set. A large accuracy drop implicates the artefact; almost no drop means either the map exaggerated its role or the model has redundant cues for the same shortcut. - **Transplant it.** Composite the marker onto images of the other class and measure how far the predicted probability shifts. A model that flips class because of a pasted mark has settled the question. - **Test the shortcut directly.** Train a small model to predict the label from the artefact region alone, and another to predict the site or device from the whole image. High accuracy on either quantifies how much label information provenance carries. - **Audit the association.** Cross-tabulate label against site, device and acquisition date. This is often faster than any of the above and frequently ends the investigation. ## Step three: fix the evaluation before the model The most important finding here is not about the network. A random split leaks provenance across train and test, so the held-out score measures the shortcut's reliability rather than the model's generalisation. Replace it: - **Group the split by site, device and patient**, holding entire groups out, so the evaluation asks the deployment question: does this work on equipment it has not seen? - **Report stratified metrics** per site and per device, not just the pooled number. A large spread across sites is the signature of a provenance-driven model even when the aggregate looks excellent. - **Keep an external evaluation set** from an institution that contributed nothing to training, and treat it as the number that governs release. Expect the headline metric to fall. That drop is not a regression; it is the previous number being corrected. ## Step four: remedies, in order of preference 1. **Fix the data.** Collect or rebalance so that acquisition source is not correlated with the label. Slow, expensive, and the only fix that removes the cause rather than the symptom. 2. **Remove the artefact.** Crop the border region or mask known marker locations consistently at training and inference. Cheap and effective for one artefact — and it does not touch the underlying provenance correlation, so the model may simply migrate to a subtler proxy such as exposure, field of view or noise characteristics. Re-audit after the fix rather than assuming it worked. 3. **Augment against it.** Randomise or synthesise markers, borders and contrast so the cue becomes uninformative during training. 4. **Penalise the shortcut.** Train so that the representation cannot predict the site or device, keeping provenance labels for exactly this purpose. ## The organisational call The ship decision is straightforward: do not release on a metric you have just learned is measuring the wrong thing. The leadership question is what this incident changes going forward. Provenance metadata becomes mandatory at ingestion, grouped splitting becomes the default rather than something a careful engineer remembers, and an attribution audit over a sample of predictions becomes a release gate with its findings recorded in the model documentation. Also set expectations with clinical stakeholders in advance: a heatmap is an internal debugging instrument that can reveal a shortcut, not a per-patient justification to display at the point of care — its resolution is too coarse and its faithfulness too method-dependent to carry that weight.