How do you set the patience for early stopping when the validation curve is jagged?

### What patience actually buys Early stopping halts after some number of consecutive validation checks without improvement. That number is `patience`, and it exists for one reason: the validation curve is a *sample statistic*, not a smooth function. It wobbles because the validation slice is finite, because the optimizer path is stochastic, and because the metric itself may be coarse. Patience of one halts on the first wobble; a well-chosen patience halts only when the wobble has lasted long enough to be evidence. ### Reading the curve first Consider a validation log-loss that reads 0.462, 0.451, 0.455, 0.458, 0.452, 0.440, 0.437 over consecutive checks. Three consecutive checks are worse than the 0.451 at check two, and then the fit finds a genuinely better region. Patience one would have halted at 0.455 and restored the 0.451 iterate — leaving real accuracy on the table for no reason other than impatience. Patience four survives the plateau and lands on 0.437. So the calibration question is empirical: **how long a run of worsening checks does noise alone produce on this curve?** You can read that off a single long diagnostic run — fit past the point you would normally stop, plot the monitored metric per check, and look at the length of the false dips. Set patience above the longest of those runs. On a smooth, well-sampled curve that might be 3; on a small validation slice or a coarse business metric it might be 20. ### The asymmetry that should drive the decision The two errors are not symmetric. - **Patience too small**: you stop before the fit is done and ship a genuinely worse model. The loss is in accuracy, and you never see it — the run looks like it converged. - **Patience too large**: you spend extra compute after the best iterate has already been passed. Because the best iterate is snapshotted and restored, the shipped model is *unaffected*. The costs are compute versus quality, and compute is usually cheaper. When in doubt, err large. This also means the common ritual of copying patience 10 from the last project is defensible as a default but indefensible as an answer — it is only right if the curve's noise scale happens to match. A useful companion knob is a minimum improvement threshold: require a new best to beat the old one by some small margin before it resets the counter. Without it, a curve that improves by 0.00001 per check will never trigger a halt, and patience does nothing. ### Patience is counted in checks, not passes If you evaluate every fifth pass, patience 3 means fifteen passes of no improvement, not three. Mixing the two units is a real and common bug: someone moves the evaluation to every fifth pass to save time and unknowingly makes the stopping rule five times more lenient. State the cadence and the patience together, and reason about the product. Cadence itself is a tradeoff. Frequent checks give a fine-grained curve and a tight stop, but each check costs a pass over the validation slice. Infrequent checks are cheap but coarse — you may overshoot the true minimum by several passes, and each check is a noisier draw of the metric, which argues for a *larger* patience, not a smaller one. ### What you monitor changes everything Monitoring validation log-loss every pass gives a smooth, sensitive signal, and it is the right default for a probability model. Monitoring a business metric — captured revenue at a fixed impression budget, say — is more faithful to the decision the model serves but is typically a step function of the threshold, evaluated on fewer effective samples, and far noisier per check. If you stop on it, raise patience and check less often; if the metric is so noisy that its check-to-check variation swamps the real improvement, stopping on it is close to random and you should stop on the smooth loss while *reporting* the business metric. ### The slice you carved out A validation slice used purely to watch the run is rows removed from training. On a few million ad impressions that is a rounding error. On 2,000 rows it is a real cost, and the stopping point chosen on 400 held-out rows is itself high variance. The fix is to spend folds instead of a single split: run the fit on each fold, record the best iterate per fold, and refit on all rows for the averaged count. You pay in compute and get back both the rows and a more stable stopping point. Finally, the score at the halt is the minimum over every check you made — a selected value, and therefore optimistic. Never quote it as the expected production number; quote a split that had no part in the stopping decision.