Maximum-Margin Classifier

## What a support vector is A hard-margin support vector machine looks for the hyperplane that maximises the distance to the nearest training point of either class. Writing the surface as `w.x + b = 0` and labels as +1 / -1, the fitted solution satisfies `y_i * (w.x_i + b) >= 1` for every training point, under the usual convention that the closest points are pinned at exactly 1. The points that attain that equality - the ones sitting exactly on the margin boundary, on the kerb of the street - are the **support vectors**. Everything else has `y_i * (w.x_i + b) > 1` strictly: it is inside its own territory with slack to spare. ## Binding and non-binding constraints The geometric statement has an optimisation statement behind it. Each training point contributes one inequality constraint. A constraint that holds with equality at the optimum is *active*, or binding: it is pressing on the solution, and relaxing or removing it would let the objective improve. A constraint that holds strictly is *inactive*: the optimum is nowhere near it, so deleting it changes nothing about where the optimum sits. Support vectors are exactly the active constraints. All other rows are inactive. Removing an inactive constraint from an optimisation problem leaves the optimum untouched - not approximately, exactly. This is why the fitted model is unchanged. ## The 5,000-row picture Suppose you fit on 5,000 labelled rows and discover that 12 of them are support vectors. Then: - Deleting any single one of the other 4,988 rows and refitting gives the identical hyperplane, to the last bit. - Deleting all 4,988 at once and refitting on just those 12 rows still gives the identical hyperplane, because those 12 constraints alone define the same widest street. - Prediction time only ever needs the fitted `w` and `b`, and in the dual view only those 12 rows carry nonzero weight - hence the model's famous compactness. ## The other direction Symmetry does not hold. Delete or move a support vector and you have removed a binding constraint, so the street can usually widen or tilt: the boundary moves. That is the real substance of the sensitivity, and it is why one should not be complacent about a model whose entire boundary rests on twelve rows. Conversely, moving a non-support vector *within its own side* changes nothing at all - until it moves close enough to become the new closest point, at which point it becomes a support vector itself and starts constraining the solution. ## Why the count is informative The number of support vectors is a rough complexity signal. A classic result bounds the leave-one-out error rate of a hard-margin machine by the number of support vectors divided by the number of training points: intuitively, leaving out a non-support vector cannot change the model, so it cannot be misclassified by the refit unless it already was. Twelve support vectors out of 5,000 therefore corresponds to a very small leave-one-out bound - the data is separated by an unusually clean, wide gap. A model where most rows are support vectors is the opposite signal: the classes crowd the frontier, and the boundary is held in place by nearly everything. ## What candidates get wrong The most common error is saying support vectors are the points *closest to the other class's centre*, or the points that were misclassified. In the hard-margin setting nothing is misclassified; support vectors are simply the points at the minimum distance from the surface. Another error is assuming that because most rows are ignorable, you may safely subsample the training data up front - you cannot, because you do not know which rows are support vectors until you have fitted on all of them, and subsampling can discard exactly the points that define the boundary.