Why is logistic regression trained with cross-entropy loss rather than squared error?

## What the objective is A sigmoid classifier predicts `p = sigmoid(w*x + b)`, a number in (0, 1). Training means choosing the weights that make the observed labels most plausible. Write the model's assumption explicitly: each row's label is a Bernoulli draw with success probability `p`. The probability the model assigns to a single observed label `y` is then ``` P(y | x) = p^y * (1 - p)^(1 - y) ``` which reads as `p` when `y = 1` and `1 - p` when `y = 0`. Taking the logarithm and negating gives the per-row loss: ``` loss = -[ y*log(p) + (1 - y)*log(1 - p) ] ``` That is cross-entropy, also called the log loss or binary cross-entropy. Summed (or averaged) over rows it is the objective the fit minimises. Because the log turns the product over rows into a sum, minimising it is the same as maximising the likelihood of the data under the model. Only one of the two terms is ever live: for a positive row the loss is `-log(p)`, for a negative row it is `-log(1 - p)`. In both cases the loss is 0 when the model is perfectly right and rises without bound as the prediction moves towards the wrong end. Predicting 0.5 for everything costs `-log(0.5) = 0.693` per row - the reference number a warranty-claim model starts from when it has learned nothing beyond a balanced base rate, and the number it must beat as the weights move. ## Why not squared error Squared error, `(p - y)^2`, is the natural reflex from linear regression, and it is a bad fit here for three separate reasons. **1. The gradient vanishes where the model is most wrong.** Differentiate both losses with respect to the linear score `z`, using `dp/dz = p*(1 - p)`: ``` cross-entropy: d loss / dz = p - y squared error: d loss / dz = 2*(p - y)*p*(1 - p) ``` The cross-entropy derivation cancels the sigmoid's slope against the `1/p` from the logarithm, leaving the residual alone. Squared error keeps `p*(1 - p)` as a multiplier. Now take a row that the model scores at 0.999 whose true label is 0 - a confidently wrong prediction, exactly the row you most want to fix. Cross-entropy gives a loss of 6.9 and a gradient of 0.999. Squared error gives a loss of 0.998 - almost its maximum of 1, so it *knows* the row is wrong - but a gradient of about 0.002, because `p*(1 - p) = 0.001` at that point. The update is roughly 500 times smaller. The model is stuck in the saturated tail and crawls out of it. **2. It is non-convex in the weights.** Cross-entropy composed with a sigmoid on a linear score is convex in `w` and `b`: there is a single optimum and no local minima to get trapped in. Squared error composed with a sigmoid is not convex, so the surface can have flat regions and local minima that depend on where you started. For a model whose whole selling point is a reliable, reproducible fit, that is a real loss. **3. It is not the likelihood.** Squared error is the maximum-likelihood objective for Gaussian noise on a continuous target. The target here is a 0/1 label, and the model's noise assumption is Bernoulli. Fitting with squared error means optimising a criterion the model does not correspond to, and forfeits the statistical machinery that follows from a likelihood. ## The gradient in weight space Applying the chain rule one more step, for weight `j`: ``` d loss / d w_j = (p - y) * x_j ``` Summed over rows this is the whole gradient, and it has a satisfying reading: each row pushes each weight in proportion to its own feature value times how far the prediction missed. A row the model already gets right contributes almost nothing; a badly missed row dominates. This form is why logistic regression trains stably with plain gradient descent, and why the update rule looks identical to the linear-regression one despite the non-linear squashing in between. ## Asymmetry, and the price of confidence Cross-entropy is unbounded above. A single row predicted at 0.001 whose label is 1 contributes `-log(0.001) = 6.9`; predicted at 0.000001 it contributes 13.8. Squared error caps out at 1 per row no matter how wrong. That unboundedness is the intended behaviour - it makes the fit strongly averse to confident mistakes and, in the tails, it is the only thing keeping the optimiser awake. The practical consequence is that a handful of mislabelled rows can dominate the objective, so if labels are noisy the objective will happily distort the fit to accommodate them. ## Common mistakes - Saying squared error "doesn't notice" a confidently wrong row. It does - the loss is near its maximum. What is missing is the *gradient*, not the loss value. - Saying cross-entropy is preferred because it is a better report of model quality. Here it is the objective the fit minimises; whether it is also the right number to report is a separate matter. - Writing the loss as `-log(p)` for every row. That term applies only to positive rows; negative rows use `-log(1 - p)`.