What should early stopping monitor when validation loss and the shipped horizon-7 MAPE disagree?

## The monitor is a statement of objective Whatever quantity early stopping watches becomes, in practice, the thing the whole run is optimized for: it decides when training halts and which weights get exported. Choosing it is therefore not a tuning detail. It is the moment where a team either aligns training with what the model is judged on, or quietly optimizes a proxy. The canonical clash: a demand-forecasting model whose validation loss keeps falling smoothly for sixty epochs, while mean absolute percentage error at forecast horizon 7 — the number the business reads — bottomed out at epoch 22 and has been drifting up since. Both curves are computed on the same held-out data, from the same predictions. They disagree because they weigh errors differently: a squared-error loss is dominated by large-volume series with big absolute residuals, while a percentage error is dominated by low-volume series where a small absolute miss is a large relative one. The model is genuinely getting better at one thing and worse at another. ## Default: monitor the decision metric If the metric you are judged on can be computed on a validation split every epoch, monitor it. Stopping on the surrogate when the two diverge means shipping a model you have already measured to be worse at its job, which is very hard to defend after the fact. The loss keeps two legitimate roles. It is what gradients are computed from — the monitor never enters the backward pass, and swapping the monitor changes nothing about optimization. And it is a lower-variance early-warning signal: a loss that stops improving at all is usually a training problem, whereas a metric that wobbles may just be wobbling. ## Moving the monitor changes the bookkeeping Three things travel with the choice, and interviewers check whether you name them. **Direction.** Loss and MAPE improve by going *down*; accuracy, macro-F1 and AUC improve by going *up*. The comparison in the stopping rule has to be set per metric. Getting it backwards is a real and embarrassingly common bug: the counter never resets, the run stops at `patience` epochs, and the checkpoint holds epoch-one weights. The symptom is a suspiciously short run whose "best" epoch is near the start. **Scale of a meaningful improvement.** A minimum-improvement threshold has to be expressed in the metric's own units — tenths of a percentage point for MAPE, not the same absolute number you used for a loss in nats. Set to zero, any hundredth-of-a-point flutter resets the counter and the run never stops; set too large, everything looks like a plateau. **Noise.** Decision metrics are usually noisier than the loss because they are coarser. Consider monitoring macro-F1 on a 12-class imbalanced validation set: macro averaging gives a class with thirty validation examples the same weight as one with three thousand, so two flipped predictions can move the headline by a point. The loss, by contrast, moves continuously with every probability shift. A monitor with a wider noise band needs a longer patience and a threshold sized against that band, or you are stopping on noise. If a rare class is the source of most of the jitter, a per-class floor as a guardrail alongside a steadier primary metric is often better than monitoring the volatile average directly. ## Decoupling the two rules Since stopping and selection are separate mechanisms, they may watch different quantities. A common and defensible arrangement: quit when the smooth validation loss stops improving for a generous patience, but write the exported checkpoint whenever the decision metric hits a new best. You get the stability of the loss for the halting decision and the alignment of the metric for the artifact you ship. The cost is a run that may keep going after the metric has clearly turned — cheap when epochs are cheap, wasteful when they are not. ## The judgment a lead owns - **One primary, the rest as guardrails.** Several metrics matter, but only one can drive a counter. Name a primary and express the others as thresholds a candidate checkpoint must clear. - **Selection is fitting.** Every epoch you compare on the validation split is a small fit to that split. A noisy metric on a small validation set scanned over sixty epochs can be selected almost entirely on luck. Keep a test split untouched by both rules, and be suspicious when the selected metric and the test metric drift apart across runs. - **Decide once, write it down.** If different runs stop on different quantities, their scores are not comparable and no sweep or A/B between them means anything. - **Stability over sensitivity when the metric is thresholded.** A metric that depends on a decision threshold moves for two reasons — the ranking got better, or the threshold drifted relative to the score distribution. Monitoring a threshold-free quantity and tuning the threshold once at the end separates those.