Why does a soft-margin SVM minimise hinge loss instead of the 0/1 error rate?

## Two losses, side by side Code the label as `y = +1` or `y = -1` and let `f(x) = w . x + b` be the raw, unsquashed score. Define the **margin of a point** as `m = y * f(x)`: positive when the prediction is correct, and larger the more confidently correct it is. - **0/1 loss**: `1` if `m < 0`, else `0`. This is exactly what accuracy measures. - **Hinge loss**: `max(0, 1 - m)`. Zero once `m >= 1`, then rising linearly at slope 1 as `m` falls. Evaluate hinge on a few points to feel its shape. `m = 2.3` gives `0`. `m = 1.0` gives `0` — the point sits exactly on the margin. `m = 0.4` gives `0.6` — correctly classified, but inside the margin. `m = -0.8` gives `1.8` — misclassified, and charged more the deeper it is on the wrong side. ## Why the error rate cannot be the training objective The 0/1 loss is piecewise constant. Nudge `w` a little and either nothing changes or one point flips and the total jumps by a whole unit. Its derivative is zero everywhere it exists and undefined at the jumps, so a gradient tells the optimiser nothing about which direction reduces error. The summed 0/1 loss over a linear model is also non-convex with many flat plateaus and, in general, minimising it exactly is computationally intractable — there is no efficient algorithm that finds the error-minimising hyperplane on arbitrary non-separable data. The standard escape is a **convex surrogate**: replace the loss you care about with a convex function that sits above it, minimise the surrogate, and inherit a bound on the thing you wanted. Hinge loss qualifies. It is convex (a maximum of two affine functions), and it dominates the 0/1 loss everywhere: at `m = 0` the 0/1 loss is at most 1 and hinge is exactly 1; for `m < 0` hinge is `1 - m > 1`; for `0 < m < 1` hinge is positive while 0/1 is zero. So the average hinge loss upper-bounds the training error rate, and driving it down squeezes the error rate from above. That also explains the residual gap: the two objectives do not have the same minimiser. A model can trade one deep violation for several shallow ones, lowering total hinge while leaving the misclassification count flat or even slightly worse. ## The unit of margin, and why hinge is not just "relu of the error" Hinge charges nothing only when `m >= 1`, not when `m >= 0`. That `1` is the whole point: it demands a *buffer*, not merely a correct sign. Because `w` can be rescaled freely, fixing the required margin at 1 and letting `||w||` float is what turns "be confidently right" into the concrete penalty `0.5*||w||^2`. The full objective ``` 0.5*||w||^2 + C * sum_i max(0, 1 - y_i * f(x_i)) ``` is therefore an ordinary regularised empirical-risk problem: an L2 penalty on the weights plus a convex data-fitting loss, with `C` weighting the second against the first. The slack variables in the constrained form are not an extra modelling choice — at the optimum, `slack_i` equals exactly `max(0, 1 - m_i)`, the point's hinge loss. Three regimes follow directly: `slack = 0` means the point is on or beyond the margin; `0 < slack <= 1` means it is inside the margin but still on the correct side; `slack > 1` means it is misclassified. ## Consequences of the flat region The zero region is where hinge earns its keep. A point with `m = 3` contributes zero to the objective and zero to the subgradient. Delete it from the training set and the solution does not move. Only points with `m <= 1` — those on or inside the margin — determine the fitted weights, so the solution is **sparse in the training data**. This is a genuine difference from losses that stay strictly positive for every point and therefore let every observation, however comfortably classified, exert a pull on the boundary. The cost is a kink at `m = 1`, where hinge is continuous but not differentiable. Solvers handle this with a subgradient: any value in `[-1, 0]` times `y*x` works at the kink, and the objective remains convex, so convergence guarantees survive. ## What hinge loss does not give you It is not a proper scoring rule. Because it saturates at zero, it stops caring how confidently a point is classified once it clears the margin, so the raw score is a signed distance-like quantity, not a probability. If a calibrated probability is required, fit a monotone map from score to probability on held-out data as a separate step; reading the raw score as a probability is a common and costly mistake. Hinge also grows only linearly in the violation, so a single far-out mislabelled point is charged proportionally rather than quadratically. That makes it less explosive than a squared penalty on the same violation, but it is still unbounded: one badly wrong label placed far across the boundary can still drag the solution, especially at large `C`.