Why does a raw input-gradient saliency map for an image classifier look like speckle noise?

## What the map actually is Input-gradient saliency, the oldest gradient attribution map, is just the partial derivative of a chosen class's score with respect to each input value, displayed as an image (usually as absolute value, or as the maximum absolute value across colour channels). The claim it encodes is narrow and first-order: *if this pixel were nudged infinitesimally, the class score would change at this rate*. ## Why the map is noisy **It is a point estimate of a rapidly varying quantity.** A network built from rectified linear units is a piecewise-linear function of its input. Each unit's on/off pattern defines a linear region, and the number of such regions is astronomically large, so regions are tiny. Inside a region the gradient is exactly constant; step across a boundary and it changes discontinuously. The gradient at one specific image is therefore a sample of a function that varies on a scale far smaller than any visually meaningful perturbation. Two images a human cannot tell apart can produce visibly different saliency maps. **Local slope is not the same as contribution.** A pixel that is essential to the prediction can have a near-zero derivative because the score has saturated: if the evidence is already overwhelming, marginal changes do not move the output. Conversely, a pixel that contributes nothing can sit on a steep local slope. The gradient measures sensitivity in an infinitesimal neighbourhood, not how much of the decision that pixel is responsible for. **Display choices amplify it.** The distribution of gradient magnitudes is extremely heavy-tailed. A handful of pixels dominate the scale, so a linear colour map shows a few bright specks on black. Practitioners clip at a high percentile — say the 99th — before normalising, which is a presentation fix, not a faithfulness fix, and it can make an unstable map look decisive. ## Making it readable **Average over noisy copies (SmoothGrad).** Draw `n` samples `x + e` with `e` Gaussian, standard deviation set as a fraction of the input's value range (a common working range is roughly ten to twenty percent of the range), compute the saliency map for each, and average. This approximates the gradient of a locally smoothed version of the score function, replacing an arbitrary point sample with a neighbourhood estimate. Visual quality improves quickly for the first few dozen samples and then plateaus. The two knobs interact: too little noise and nothing changes; too much and you are explaining a different, blurrier image. **Aggregate along a path instead of at a point.** Integrating the gradient along a path from a reference input to the actual input replaces the single local slope with an accumulation, which sidesteps the saturation problem — that is what integrated gradients does, at the cost of having to choose a reference. **Move up a level of abstraction.** Attributing at a convolutional layer's feature maps rather than at pixels produces a coarse but stable map, because the spatial averaging in the construction throws away exactly the high-frequency component that makes pixel maps unreadable. **Stop using gradients.** Occlusion attribution slides a patch of neutral value across the input and records how far the predicted class probability falls at each position. It is a finite, human-scale perturbation rather than an infinitesimal one, so it answers a question you can actually verify, and it is model-agnostic. The price is one forward pass per patch position and a patch-size tradeoff: small patches give fine maps but a robust model shrugs them off; large patches give smooth maps that cannot localise anything and may cover two objects at once. ## The judgment part The most common failure here is treating visual cleanliness as evidence of correctness. Smoothing is a variance-reduction trick applied to an estimator whose bias you have not measured; a cleaner picture can be a *more* confident lie. Two disciplines protect you. First, agreement: if a smoothed gradient map, an integrated-gradient map and an occlusion map all point at the same region, the region is probably real; if they disagree, believe none of them yet. Second, verification by intervention: mask the highlighted region and measure how much the class probability actually drops. That converts an attribution picture into a measurement. One last practical note: because the map is a derivative with respect to the *input as the model sees it*, it lives in the model's preprocessed input space. If the display pipeline resizes, crops or normalises differently from the training pipeline, the map you are staring at is not aligned with the image you think it explains — a surprisingly common source of maps that look nonsensical for a completely mundane reason.