What does hinge loss's margin do that cross-entropy on the same scores does not?

## The functional form Write the classifier's raw score as `s` and the label as `y` in {-1, +1}. The quantity `m = y*s` is the **functional margin**: positive when the prediction is correct, negative when it is wrong, and larger when the classifier is more emphatic. Hinge loss is `L_hinge = max(0, 1 - m)` Read it as three regimes. When `m < 0` the example is misclassified and the loss is above 1, growing linearly the more wrong it gets. When `0 <= m < 1` the example is **correct but inside the margin** and still pays `1 - m`. When `m >= 1` the loss is exactly 0. The number 1 is not magic. It fixes a scale, and any other constant would be absorbed by rescaling the weights — what matters is that the loss insists on a *gap*, not merely a correct sign. ## What the flat region buys you The subgradient of hinge with respect to the score is `-y` when `m < 1` and `0` when `m > 1` (either value is admissible at the kink `m = 1`). So examples comfortably on the right side contribute **nothing** to the update. Two consequences follow: - **Sparse gradients.** Only the violating examples — the misclassified ones and the ones sitting inside the margin — shape the decision boundary. This is exactly the support-vector intuition: the boundary is determined by the hard cases, and moving a distant, easy example does not move the boundary at all. - **A natural stopping point per example.** Once an example is comfortably right, the optimiser turns its attention elsewhere. Combined with weight decay, this is what makes the margin a capacity control rather than a mere training trick: the loss pays for margin, decay pays for weight size, and the ratio between them sets how wide a margin the model is willing to buy. ## Contrast with cross-entropy The logistic form `L_log = log(1 + exp(-m))` is positive for every finite `m`. At `m = 3` it is roughly 0.049; at `m = 6` roughly 0.0025. Small, but never zero, and the gradient shrinks in proportion. Cross-entropy therefore keeps nudging an already-correct example to be *more* correct, and given a separable dataset and no regularisation it will drive the score gap toward infinity. In practice that shows up as increasingly peaked output probabilities and a validation loss that rises long after the decision boundary has settled. Which behaviour you want depends on what you need out of the model: - If you need **probabilities** — to threshold at a chosen operating point, to feed into a downstream expected-value calculation, or to combine with another model's score — cross-entropy is the right objective, because it is a proper scoring rule whose minimiser is the true conditional probability. A hinge-trained score is an arbitrary real number; it orders examples but has no probabilistic reading, and would need a separate post-hoc mapping before you could treat it as one. - If you need a **crisp decision boundary** and want the fit driven by the hard cases, hinge is attractive, and the flat region makes it robust to easy examples whose labels you would rather not over-fit. ## Squared hinge Squared hinge is `max(0, 1 - m)^2`. Two things change. First, the kink at `m = 1` disappears: the function is now continuously differentiable, which some optimisation methods prefer. Second, and more consequentially, the penalty for a badly violated margin grows quadratically instead of linearly, so the gradient magnitude is proportional to the violation rather than constant. A single mislabelled example sitting at `m = -8` now contributes a gradient about eight times larger than one at `m = -1`, where plain hinge would weight them equally. That is a genuine trade: squared hinge converges more smoothly on clean data and concentrates effort on the worst violators, but it is markedly more sensitive to label noise and outliers. On a noisy real-world dataset plain hinge is usually the safer default; the square is a choice you make when you trust your labels. ## The multi-class case The margin idea generalises. With one score per class, a multi-class margin loss penalises `max(0, 1 + s_j - s_y)` for each wrong class `j`, summed or maximised over `j`, where `s_y` is the true class's score. The demand is the same: the correct class must beat every competitor by at least the margin, not merely beat it. ## What interviewers are checking The answer they want is not the formula — it is the recognition that *correct* and *done* are different states, that hinge encodes the difference and cross-entropy does not, and that this shows up concretely as which examples still produce gradients late in training.