How does label smoothing change the target that a classifier's cross-entropy loss fits?

## The target a one-hot loss fits A `K`-class classifier ends in `K` logits that a softmax turns into a probability vector `p`. Training minimises the cross-entropy between `p` and a target distribution `q`, and with an ordinary hard label `q` is one-hot: 1.0 on the observed class `y`, 0.0 on the other `K - 1`. That target is never actually reachable. A softmax output is a ratio of positive exponentials, so no finite set of logits produces exactly 1.0 and exactly 0.0. On training data the network can separate, the loss therefore keeps falling for as long as you keep pushing `z_y` further above the other logits — there is no finite minimum, only a direction of continuing improvement. The visible symptoms are familiar: training confidence saturates near 1.0, the last layer's weight norm keeps creeping up, and the model reports 0.99 on examples it has no business being sure about. ## What smoothing changes Label smoothing changes the target, not the architecture, not the optimizer, and not what happens at inference. The smoothed target is a mixture of the one-hot vector and the uniform distribution: `q' = (1 - epsilon) * onehot(y) + epsilon * uniform` Component-wise that is `q'_y = 1 - epsilon + epsilon/K` for the labelled class and `q'_k = epsilon/K` for every other class. With `K = 10` and `epsilon = 0.1`, the uniform component contributes 0.01 to all ten entries, so the target vector reads 0.91 on the label and 0.01 on each of the nine others. Be aware of a second convention in common use: the true class gets `1 - epsilon` and the remaining `epsilon` is split over the other `K - 1` classes, giving 0.90 and 0.0111 at the same settings. The two are numerically close and behave the same way, but they are not identical, so state which one you mean when you quote a number. ## Why the logit gap becomes finite Cross-entropy against a fixed target `q'` is minimised, over all valid probability vectors, exactly when `p = q'`. Softmax depends only on logit differences, and `p_y / p_k = exp(z_y - z_k)`. Setting `p = q'` therefore pins the gap the model is trying to reach: `z_y - z_k = log( (1 - epsilon + epsilon/K) / (epsilon/K) )` At `K = 10, epsilon = 0.1` that is `log(91)`, roughly 4.51. Once the network reaches that margin the loss stops rewarding a bigger one; pushing further actually increases the loss. Under a one-hot target the same ratio is `p_y / (1 - p_y)` divided over the wrong classes, which diverges: at 0.99 the gap is about 6.8, at 0.999 about 9.1, at 0.9999 about 11.4, and it never stops. That is the honest one-line description of the method: it converts an objective with an optimum at infinity into one with an optimum at roughly 4.5. It sits alongside penalties that shrink parameters as a way of bounding how extreme the fitted solution can become, but it does it by editing the target rather than by penalising the weights. ## What it buys and what it costs Buys: measurably less extreme confidence, a small but repeatedly reported top-1 accuracy gain on large classification and sequence tasks, and some tolerance of mislabelled examples — a wrong label costs less when the target already concedes probability mass to the other classes. Costs: held-out log-likelihood and per-token perplexity get worse by construction, because the model is being trained never to put full mass on the observed class and those metrics score exactly that. Anything downstream that consumes raw probabilities — a decision threshold, a cascade's abstain rule — has to be re-tuned against the new scale. And because the loss also shapes the hidden representation, the penultimate features change, which matters if some other system reuses them. ## Practical notes - `epsilon` in the 0.05 to 0.2 range covers most reported use; larger values are chosen when labels are known to be unreliable. - Smoothing applies to the training targets only. Evaluation uses the real labels, and inference is unchanged — there is no epsilon at serving time. - The same idea applies to a single-sigmoid binary head: train toward `epsilon` and `1 - epsilon` instead of 0 and 1. - It does not apply to regression targets; there is no probability simplex to soften. - At the optimum it does not change which class wins, because the mass it adds is symmetric across the wrong classes. It still changes the learned function, since it changes every update along the way, so measured accuracy does move.