What is the difference between the functional margin and the geometric margin of a separating hyperplane?

## Two quantities that look similar Write a linear decision surface as `w.x + b = 0` and let labels be `y` in {-1, +1}. For a training point `(x_i, y_i)`: - **Functional margin**: `f_i = y_i * (w.x_i + b)`. It is positive exactly when the point is on the correct side, and it grows with the raw score. - **Geometric margin**: `g_i = y_i * (w.x_i + b) / ||w|| = f_i / ||w||`, where `||w||` is the Euclidean length of the normal vector. This is the actual perpendicular distance from the point to the surface, measured in the units of the feature space. For the dataset as a whole, each margin is defined as the minimum over all training points: the closest point sets the number. ## The rescaling test The crucial fact is that `(w, b)` and `(10w, 10b)` describe the *same* hyperplane: `10 * (w.x + b) = 0` has exactly the same solution set as `w.x + b = 0`, and the predicted sign is unchanged for every input. Now compare margins under that rescaling: - every functional margin is multiplied by 10; - every geometric margin is unchanged, because both the numerator and `||w||` scale by 10 and the factor cancels. That is the whole distinction. The functional margin is a property of the *parameterisation*; the geometric margin is a property of the *geometry*. No point moved, no prediction changed, so any quantity that changed cannot be a real measure of separation. ## Why this sinks the naive objective A first attempt at a maximum-margin rule might be "choose `w, b` to maximise the smallest functional margin". That problem has no solution: take any separating hyperplane and scale its parameters up without bound, and the objective goes to infinity while the classifier is literally unchanged. The objective is unbounded for a degenerate reason. Maximising the smallest *geometric* margin is the well-posed version, since it is invariant to that rescaling. ## Canonical scaling The invariance also gives a free choice: since scaling does not change the classifier, you may fix it however is convenient. The standard convention is the **canonical form** - normalise so the closest points have functional margin exactly 1, i.e. impose ``` y_i * (w.x_i + b) >= 1 for every i ``` with equality for the closest points. Under this convention: - the geometric margin of the dataset is `1 / ||w||`; - the two supporting hyperplanes `w.x + b = +1` and `w.x + b = -1` sit one on each side, and the perpendicular gap between them - the width of the street - is `2 / ||w||`; - maximising `2 / ||w||` is the same as minimising `||w||`, and by convention `||w||^2 / 2`, which is smooth and convex. So the familiar objective, "minimise the squared norm of the weight vector subject to every point having functional margin at least one", is not an arbitrary formula: it is the geometric-margin maximisation problem after the scale ambiguity has been removed. ## Reading the two numbers in practice Because the functional margin depends on the parameterisation, its absolute value tells you nothing on its own - a score of 4 from one fitted model and a score of 0.4 from another may describe identical boundaries with identical confidence. Comparisons are only meaningful after normalising by `||w||`, or when comparing points under one fixed model. Under the canonical convention the two coincide in a useful way: a point with functional margin 1 sits exactly on the kerb, functional margin greater than 1 means it is off the street entirely, and in the strictly separable case no point has functional margin below 1. ## Common confusions A frequent error is treating a large raw score as evidence of a confident classifier without dividing by `||w||`. Another is thinking the geometric margin is unitless: it is a distance in feature space, so it carries whatever units the features are measured in, and comparing margins across differently-measured feature sets is not meaningful. A third is assuming shrinking `||w||` makes the boundary move - it does not move the surface, it widens the street, which is precisely why the optimisation targets it.