Why does squared error on a softmax classification head train so slowly?

## The setup You have a classification head: logits `z`, probabilities `p = softmax(z)`, and a one-hot target `y`. Two candidate objectives: ``` cross-entropy: L_ce = -log p_c (c = index of the true class) squared error: L_mse = 0.5 * sum_k (p_k - y_k)^2 ``` Both are minimised by `p = y`, so at first glance the choice looks cosmetic. It is not, because training does not care where the minimum is - it cares about the gradient everywhere else. ## Where the gradient goes The softmax Jacobian is `dp_k/dz_m = p_k * (delta_km - p_m)`. Chain squared error through it: ``` dL_mse/dz_m = sum_k (p_k - y_k) * p_k * (delta_km - p_m) ``` Every single term is multiplied by a probability `p_k` coming from the Jacobian, and often by a second probability from the `(delta_km - p_m)` factor. Now take the case that matters: the model is badly wrong, with the true class at `p_c = 0.01`. The error `(p_c - 1)` is large in magnitude, about `-0.99`, which is what you want. But it is immediately multiplied by `p_c = 0.01` and then by factors like `(1 - p_c)`, so the surviving signal is of order `10^-2`, and the cross terms are of order `10^-4`. The example the model most needs to learn from produces one of the smallest updates in the batch. Cross-entropy inverts that relationship. `d(-log p_c)/dp_c = -1/p_c` is the exact reciprocal of the leading Jacobian factor, so the two cancel and the whole gradient collapses to `dL_ce/dz = p - y`. At `p_c = 0.01` the correct logit's component is `-0.99` - close to the maximum this loss can produce. Gradient magnitude grows with wrongness under cross-entropy and shrinks with wrongness under squared error. That is the entire practical story: the run does not diverge, it crawls, and it crawls worst at the start when everything is wrong. ## The non-convexity The second problem is about the shape of the objective rather than its slope. Take the simplest possible model, logits that are a linear function of the input. Cross-entropy composed with a softmax over linear logits is **convex** in the weights - a single basin, no spurious local minima. Squared error composed with the same softmax is **not** convex: the composition introduces curvature that can be flat or misleading in regions where the model is confidently wrong, and the objective admits stationary points that are not the global optimum. For a deep network the whole objective is non-convex either way, so this is not a proof that the deep case is safe with cross-entropy. But the linear case is the diagnostic: it tells you the badness comes from composing squared error with a softmax, not from the depth of the model, and every extra layer sits on top of that already-bad output geometry. ## What it looks like in practice The symptom is a run that appears to be working and is not: the loss falls a little and then flattens well above what the task should reach, accuracy hovers near the majority-class rate for a long time, and gradient norms at the output layer are orders of magnitude smaller than they should be. Nothing errors out. It is much easier to spot when you know that the flat phase is worst early - when every example has the true class near the bottom of the distribution - and eases as the model happens to get some examples right, which is the reverse of a normal training curve's dynamics. Raising the learning rate does not rescue it. The vanishing factor is **per example**: the healthy examples get the same multiplier as the sick ones, so a global step-size increase scales both and destabilises the healthy ones before it makes the stuck ones move. And a step size cannot change the geometry of a non-convex plateau. ## When squared error on probabilities is actually fine The argument above is about **class labels**. If the target is a genuine continuous proportion - a fraction of clicks, a mixing coefficient, a rate you actually measured - then it is a regression task whose output happens to lie in `[0, 1]`, and squared error against it is a defensible modelling choice with a bounded, non-explosive gradient. What you must not do is take a one-hot label, call it a regression target, and pay the vanishing-gradient tax without noticing. ## How to answer this in an interview The strong answer has three beats, in this order: (1) the gradient of squared error keeps the softmax Jacobian factor while cross-entropy's logarithm cancels it, (2) the consequence is a near-zero update precisely for the most-wrong examples, since the probability factor multiplies the error rather than the other way round, and (3) the composed objective additionally loses convexity that the cross-entropy version retains in the linear case. The weak answer stops at `cross-entropy is the standard choice for classification`, which tells the interviewer only that you have read the defaults.