Why can a network's test error worsen for dozens of epochs and then improve for hundreds more?

## The observation Hold the model, the data and the schedule fixed and plot test error against epoch. The textbook picture is a descent followed by a rise once the model starts overfitting. Epoch-wise double descent is the observation that for a sufficiently large network on data with appreciable label noise, the curve can go **down, up, and down again**: it improves early, degrades across a middle stretch - say from epoch 20 to epoch 60 - and then improves steadily for another four hundred epochs, often ending better than the pre-degradation best. Nothing about the model changes across this run. Same width, same depth, same data, same optimizer. ## Why time acts like capacity The key idea is **effective capacity**: how much of the network's expressive power the training procedure has actually recruited so far. At initialization the function is nearly featureless; after a few epochs it captures broad, high-signal structure that most examples agree on; only much later does it have the fitted detail needed to force the rare, contradictory, mislabelled examples into place. So a single long run traverses the same three regimes that a capacity sweep traverses: - **Early**: effective capacity is below what is needed to interpolate. The fit is coarse, generalization improves - this is the classical descent. - **Middle**: effective capacity approaches the amount needed to interpolate this training set. The network starts memorizing the hard and corrupted examples, and to do so with barely enough recruited capacity it distorts the function around them. Test error rises. This is the epoch-wise analogue of sitting on the interpolation threshold. - **Late**: training loss is essentially at the floor and the network is interpolating. Updates no longer trade off fitting errors; they keep moving within the space of solutions that all achieve near-zero training loss, drifting toward smoother, lower-norm ones. Test error descends a second time. ## What it means at the console This is a real diagnostic trap. You look at a run where validation error has been climbing for forty epochs, conclude the model is overfitting, kill it and shrink the model. Sometimes that is correct. Sometimes you have stopped in the middle of the bump and thrown away the best model the configuration could have produced. The distinguishing evidence, in order of cost: 1. **Is the training loss still falling steeply?** The second descent only becomes possible once training loss has essentially bottomed out. A run whose training loss is still dropping fast is nowhere near its interpolation point, and the rise is more likely to be ordinary overfitting. 2. **How noisy are the labels?** With clean, well-curated labels there is far less to memorize, and the middle bump is usually shallow or absent. If you know the corruption rate is meaningful, the bump becomes a live hypothesis. 3. **How big is the model relative to the dataset?** A small model never gets past its interpolation point at all, so it can only overfit. 4. **Run the long baseline once.** The cheapest decisive test is a single run continued far past where you would normally stop, with the same seed and schedule, to see whether the curve turns over. Do it once for the configuration family, not for every experiment. ## Interactions with the rest of the recipe A schedule that anneals the learning rate to zero quickly ends the run inside the first phase, so the second descent never gets the chance to appear. Strong explicit regularization suppresses the bump in the same way it suppresses the model-wise peak: it prevents the exact interpolation of noisy labels that creates the instability in the first place. That gives the honest framing for a senior answer. Epoch-wise double descent is not a reason to train every model for a thousand epochs. It is a reason to know **why** your validation curve is rising before you act on it, and to recognize that 'train much longer' and 'regularize properly' are two different routes out of the same bump - with very different compute bills. ## The claim to avoid Do not claim that longer training always recovers. On clean data, with a modest model, or under strong regularization, a rising validation curve after the training loss has flattened usually means exactly what it looks like. The phenomenon is conditional, and stating the conditions is most of the answer.