How does the neighbourhood kernel width change a local surrogate's explanation of the same prediction?

## What the width actually is A local surrogate weights each perturbation by its distance from the row being explained, typically with an exponential kernel: `w(z) = exp(-d(x, z)^2 / sigma^2)`, where `d` is a distance in the readable representation (for text, often how many words were deleted) and `sigma` is the **kernel width**. Large `sigma` makes the weights decay slowly, so far-away variants still count; small `sigma` makes them decay fast, so only near-identical variants count. Nothing in the data tells you what `sigma` should be. It is a modelling choice, and it is the choice the explanation is most sensitive to. ## The wide end Push `sigma` up far enough and every perturbation carries roughly equal weight. The fit is then no longer local at all; it is a sparse linear approximation of the model over a broad swathe of input space. Applied to the removed forum post, the top terms drift toward the vocabulary the model treats as risky *in general* — slurs, spam markers, common abuse patterns — even if none of them is what pushed this particular post across the line. The explanation reads plausibly, which is exactly why it is dangerous: an appeals reviewer cannot tell from the output that the answer describes the model's average behaviour rather than this decision. ## The narrow end Push `sigma` down and the **effective sample size** collapses. Formally the effective size is roughly `(sum w_i)^2 / sum w_i^2`; when almost all weight sits on twenty of your five thousand variants, you are fitting a linear model to twenty points. Two failure modes follow. First, variance: change the perturbation draw and the surviving terms change, because so few points carry the fit. Second, degeneracy: variants that differ from the original by one deleted word usually receive the *same* predicted class, so the surrogate has almost no output variation to explain and the coefficients shrink toward noise around a constant. ## Why fit quality cannot choose the width for you The tempting fix is to select `sigma` by maximising how well the surrogate reproduces the model on the weighted neighbourhood — a weighted R-squared, say. This does not work, because that quantity is not comparable across widths. As `sigma` shrinks, the weighted evaluation set concentrates on points where the model's output barely varies, and almost any linear fit reproduces a nearly constant target well. Local fidelity therefore tends to rise as the neighbourhood shrinks, all the way down to the degenerate limit. Fidelity is a useful check *at a fixed width* — it tells you whether a five-term linear model can describe that region at all — but it is not a selection criterion for the width itself. ## The sparsity knob interacts with it Width and sparsity are not independent. A wide neighbourhood contains more of the model's structure, so a five-term surrogate has to compress more and fits worse; the same five terms in a narrow neighbourhood fit almost perfectly and mean much less. When an explanation looks crisp, ask which of the two knobs bought the crispness. ## What to do in practice 1. **Fix the width by a stated rule**, not by eyeballing the output until it looks sensible — that is fitting the explanation to your prior. 2. **Record it as metadata** alongside every explanation you store or show. An explanation without its neighbourhood definition is not reproducible and not auditable. 3. **Sweep and report stability.** Recompute the explanation over a range of widths spanning roughly an order of magnitude. Terms that stay in the top set across the range are the ones worth showing; terms that appear only at one width are artefacts of the setting. 4. **Escalate when the sweep disagrees.** If the top term at a narrow width is absent at a moderate one, the honest output is 'this decision is not stably explainable by a local linear model', not the prettier of the two answers. ## The interview point What a senior answer conveys is that a local explanation is not a measurement read off the model — it is the output of a procedure with a tuning parameter that nobody validated against a ground truth. The competent response is to treat the width as a declared assumption, hold it fixed across cases so explanations are comparable to each other, and test conclusions for survival across it.