What stopping criteria end a gradient-descent fit when you watch only the training loss?

## The question being asked "When do I stop?" has two different answers, and it is worth separating them out loud. One is **numerical convergence**: iterating further does not move the weights, so more compute is wasted. The other is **when further fitting starts to hurt generalisation**. This is about the first: given only the training loss and the gradient, how do you know the optimisation is finished? Three criteria answer it, and a robust fit uses all three together. ## 1. Gradient norm below a tolerance At the minimum of a differentiable loss the gradient is zero, so a small gradient norm means the weights are near a stationary point. Stop when `||g|| < tol`. Two cautions make this a senior answer rather than a textbook one. First, the norm is not scale-free: it grows with the number of features and with the scale of the loss, so an absolute threshold like 1e-6 that works on one problem is meaningless on the next. Use a relative form — compare `||g||` to its value at the first iteration, or normalise by the number of parameters. Second, and specific to mini-batch fitting, the gradient of a *single mini-batch* does not go to zero at the optimum: individual batches disagree with the full data, so per-batch gradients stay noisy forever. If you want to threshold a gradient norm you must compute it on the full training set, typically once at the end of each epoch. ## 2. Relative improvement in the loss Stop when the loss stops improving meaningfully between epochs: `(L_prev - L_now) / max(1, |L_prev|) < 1e-6`. This is the criterion most people actually use, because the loss is already being computed and needs no extra pass. The mini-batch caveat matters here too. A single batch's loss bounces around from batch to batch for reasons that have nothing to do with progress — it reflects which rows landed in that batch. On a 10-epoch mini-batch fit over 2 million sensor rows, the per-batch loss is visibly noisy while the *epoch-average* trend is clean and monotone. So compare epoch averages, or a loss evaluated on a full pass, and never threshold a single batch. It is also worth requiring the criterion to hold for two or three consecutive epochs, so one lucky epoch does not end the run. A close variant is to threshold the **parameter change**: stop when `||w_new - w_old||` relative to `||w||` is tiny. This is sometimes preferable, because it directly measures the thing you care about — the weights have settled — and it is insensitive to the loss's scale. ## 3. A maximum-epoch budget A hard cap — say 100 epochs — is the backstop. It exists because the other two criteria can *never* fire: a step size too large makes the loss grow rather than shrink, a step size too small makes progress real but glacial, and a badly conditioned problem can crawl for a very long time. Without a cap, either case runs forever. The cap is not a convergence criterion; it is a guarantee of termination, and hitting it is a *signal* — a run that stopped on the budget rather than on a tolerance should be treated as unfinished and investigated, not shipped as if it had converged. ## Putting them together The usual policy is: run at most `max_epochs`; after each epoch compute the epoch-average training loss (and, if cheap, the full-data gradient norm); stop when either the relative loss improvement or the gradient norm falls below its tolerance for a couple of consecutive epochs; log which criterion fired. That last detail is the one people skip and then regret — six weeks later, "did that model converge or did it just run out of epochs?" is unanswerable unless the run recorded the answer. ## Things that go wrong - **Thresholding absolute loss.** "Stop when loss < 0.1" is meaningless: the achievable loss depends on the data's irreducible noise, not on the optimiser. - **Thresholding an absolute improvement on an unscaled loss.** An improvement of 1e-6 is enormous if the loss is 1e-5 and negligible if the loss is 1e6. Use the relative form. - **Stopping on the first non-improving epoch.** With mini-batch noise, one flat epoch is normal; requiring a small number of consecutive flat epochs is more stable. - **Treating a hit budget as convergence.** It means the opposite. Note finally that all of this concerns the *training* loss and the health of the optimisation. Deciding to halt a fit that is still improving on training data because it has stopped improving elsewhere is a different decision with a different justification, and should not be confused with the convergence test.