Why does gradient descent on a logistic regression's log loss reach the same fit from any starting weights?
answer
- one bowl, not a mountain range
- no local minima to escape
- zero gradient means global best
- starting weights wash out
- restarts buy you nothing
basics
~20 sLog loss is convex in a logistic model's weights, so the error surface has one global minimum and no local minima. Every run that converges lands on the same coefficients, which is why random restarts add nothing.
solid answer
~50 sWritten in terms of the linear score `z = w*x + b`, the log loss of one row is `log(1 + e^z) - y*z`, which is convex in `z`; and `z` is a linear function of the weights, so the training loss is convex in the weights too. A convex surface is a single bowl: there are no local minima that are not also global, and any point where the gradient is zero is the best fit available. So the starting point is a non-issue. Two runs from different random initialisations on the same data land on the same log loss to six decimals, and the multiple-restart habit borrowed from non-convex training buys nothing here. Linear regression's squared error is convex in its coefficients for the same structural reason. What convexity does not promise is that a finite minimum exists — only that any minimum you reach is the global one.
go deeper
Be ready to state the consequence in one line: the loss is a single bowl in the weights, so there is one best fit and the starting point does not change it. Knowing that restarts are pointless here is the whole screening answer.
Explain why the shape holds: log loss rewrites as log(1 + e^z) - y*z, which curves upward in the score, and the score is linear in the weights. Be able to say the same for squared error on a linear model.
Show where the guarantee stops. Convexity buys you a global optimum if one exists; it does not buy existence, uniqueness under duplicated features, or speed. Use a refit from a different start as a diagnostic rather than as a ritual.
Own the tradeoff of staying in the convex family at all: a reproducible, auditable fit that any two runs agree on, versus the flexibility of models whose training result depends on the run. Argue when reproducibility is worth the ceiling on accuracy.
## What "convex in the weights" means in practice When people say a training loss is convex, they mean it as a function of the model's **parameters**, with the data held fixed. Picture the loss plotted against the coefficients: a convex loss is one smooth bowl. Wherever you start on the wall of that bowl, downhill always means "toward the same bottom". A non-convex loss is a mountain range with many separate hollows, and where you end up depends on where you began. Two details matter and are often confused. Convexity is a property of the **loss plus model**, not of the algorithm you use to fit it, and not of the data being easy. And it is convexity *in the parameters* — not in the inputs, and not in the predicted probabilities. ## Why linear regression's squared error qualifies The prediction `yhat = w*x + b` is linear in the coefficients. The loss `sum (yhat - y)^2` is therefore a sum of squares of linear functions of `w`, which is a quadratic bowl in `w`. Its curvature does not depend on where you are in weight space at all — the same bowl shape everywhere. ## Why logistic regression's log loss qualifies For one row with label `y` in {0, 1} and predicted probability `p = 1 / (1 + e^-z)`, the loss is ``` loss = -[ y*log(p) + (1-y)*log(1-p) ] ``` Substituting the sigmoid and simplifying gives a compact form in the score: ``` loss = log(1 + e^z) - y*z ``` The first term is the softplus function, which curves upward everywhere; the second is linear in `z`, and adding a linear term never breaks that upward curvature. So the per-row loss is convex in `z`. Because `z = w*x + b` is linear in the weights, and summing convex pieces keeps the shape, the total training loss is convex in the weights. The Hessian works out to a weighted product of the design matrix with itself, with weights `p*(1-p) >= 0`, so it is never negatively curved in any direction. ## What convexity buys you - **Any stationary point is the answer.** If the gradient is zero, you are at a global minimum. There is no need to ask whether a better set of coefficients exists elsewhere. - **Initialisation is irrelevant to the result.** Starting all weights at zero, at small random values, or at last week's coefficients changes only how many steps you take, not where you land. - **No restart ritual.** Fitting from ten random starts and keeping the best is wasted compute here. If you do it as a sanity check, the ten results should agree to many decimal places. - **One number to report.** The fitted coefficients are a property of the data and the model, not of the run. ## What convexity does not buy you - **It does not guarantee a finite optimum exists.** If some feature combination separates the two classes perfectly, the log loss keeps falling as the weights grow, approaching zero without ever reaching it. The surface is still convex; it simply has no bottom in finite weight space, and the fit never settles. - **It does not guarantee a *unique* optimum.** If two features are exact duplicates, or more generally if the design has a flat direction, there is a whole ridge of weight vectors with identical loss. The loss value is unique; the coefficients are not. This is why two runs can agree on the loss to six decimals and still report different individual weights. - **It says nothing about speed.** A convex bowl can be extremely elongated, and a poorly scaled problem still takes many steps. - **It says nothing about model quality.** A convex loss with a terrible feature set converges beautifully to a bad model. - **It does not survive stacking.** Convexity here comes from the score being *linear* in the weights. Put hidden nonlinear layers between the inputs and the score and the loss stops being convex in the parameters — that is why non-convex training methods worry about initialisation and local minima and this model does not. ## The practical check If you suspect something is wrong with a fit, refit from a different initialisation. Identical final loss is confirmation the optimiser is behaving. Different final loss means you stopped early, your tolerance is loose, or the run never converged at all — not that you found a second optimum, because there is not one to find.
- Does convexity guarantee that the fit converges to a minimum?No. Convexity guarantees that any minimum you reach is the global one; it does not guarantee a minimum exists. If a feature combination separates the classes perfectly, the loss keeps decreasing as the weights grow, so there is nothing finite to converge to and the run stops only when it hits an iteration cap.
- Is the loss still convex if you stack hidden nonlinear layers before the score?No. The convexity here comes from the score being a linear function of the weights, with a convex loss applied on top. Insert nonlinear hidden layers and the loss becomes non-convex in the parameters, which is exactly why that setting cares about initialisation, restarts and local optima while a plain linear or logistic fit does not.
- Two runs from different starts give the same loss but different coefficients. What does that tell you?Not that there are two optima — the loss is the same, so both sit at the global minimum. It means the minimum is a flat ridge rather than a point, typically because two features are duplicates or near-duplicates. The prediction is identified; the individual coefficients are not, so do not interpret them.
A convex loss is one valley: drop a marble anywhere on the slope and it rolls to the same bottom. A non-convex loss is a mountain range full of separate hollows, and the marble stops in whichever one you dropped it into.
saying these in an interview costs you the question
- Says you need random restarts to escape local minima here
- Treats convexity as a promise of fast convergence
- Claims a convex loss always has a finite minimiser
- Says the loss is convex in the inputs rather than the weights
- Assumes any model trained with log loss is convex