In an RBF-kernel SVM, what does gamma control, and what breaks when it is large?

## The function gamma lives in The RBF (radial basis function, also called Gaussian) kernel is ``` k(x, z) = exp(-gamma * ||x - z||^2) ``` where `||x - z||` is the Euclidean distance between the two points and `gamma > 0`. Read the formula as a similarity score: it returns 1 when `x` and `z` are the same point, and decays monotonically towards 0 as they move apart. Nothing else in the expression varies, so gamma alone decides how quickly that decay happens. It is often written in the equivalent bandwidth form `exp(-||x - z||^2 / (2*sigma^2))`, which makes the relationship explicit: `gamma = 1 / (2*sigma^2)`. Large gamma is a small sigma is a narrow bell. The most useful mental label is **inverse neighbourhood radius**: gamma answers "how far away does a point stop mattering?" ## Why that becomes the shape of the boundary A trained kernel SVM scores a new point as a weighted sum of kernel values against the support vectors it kept, plus an offset. Each support vector therefore contributes a bump of influence centred on itself, and gamma is the width of that bump. - **Large gamma — narrow bumps.** A support vector's influence dies out before it reaches its neighbours. The boundary stops being a smooth surface and becomes a collection of small islands, one wrapped around each awkward training point. Every training point can be given its own island, so the training set is classified perfectly; nothing has been learned about the space between the points. The matrix of pairwise kernel values approaches the identity matrix — every point is similar only to itself — and the model has effectively memorised its data. - **Small gamma — wide bumps.** The exponent goes to zero for every pair, so every kernel value approaches 1 and all points look alike. The weighted sum becomes almost the same number everywhere, the boundary becomes very smooth and eventually almost featureless, and the model underfits: it cannot express even the structure that is genuinely there. Between those extremes is a band where the bumps overlap enough to interpolate between training points but not so much that all local structure is smeared away. ## Diagnosing it in practice A worked case: 60 spectrometer channels per sample, the task is identifying which material produced the spectrum, and you sweep gamma from 0.001 up to 100 with everything else fixed. - At `gamma = 0.001` training and validation accuracy are both mediocre and nearly equal. Both curves being low and together is the signature of underfitting — the model is too smooth. - Somewhere in the middle both rise, validation peaks, and the gap between the two stays modest. That is the band you want. - By `gamma = 100` training accuracy is 100% and validation has fallen off a cliff. A large train/validation gap with perfect training performance is the signature of the islands regime, and no amount of extra data collection fixes it while gamma stays there — the setting itself is wrong. Two practical notes follow. First, sweep gamma **logarithmically**, not linearly: the interesting range spans orders of magnitude, and a linear grid from 0.001 to 100 wastes almost every point on the high end. Second, since gamma multiplies a squared Euclidean distance, the sensible starting scale depends on the typical squared distance between points in your data, which depends on the number of features and their units — a value that worked on one dataset carries no meaning on another. ## What large gamma is not It is not a bias problem, so do not respond to the symptom by adding capacity elsewhere. It is variance: the model is too flexible and is fitting the sample rather than the population. It is also not a data-quantity problem in the usual sense — with gamma huge enough, more data simply means more islands. And gamma is a property of the kernel, not of the classifier's tolerance for misclassification; those are separate dials that must be searched together rather than one after the other, because a very local kernel and a very forgiving fit can partially mask each other's symptoms on the training set. ## The reasoning to show A good answer connects three levels in order: the formula (gamma scales the squared distance inside an exponential), the geometry (it is the width of each support vector's zone of influence), and the diagnosis (large gamma yields perfect training accuracy with collapsed validation accuracy). Anyone who can only recite "large gamma overfits" has memorised the symptom without the mechanism, and the follow-up about what the kernel matrix looks like in that regime will expose it.