Double Descent

## The shape being described Take one model family, one dataset and one training recipe, and sweep a single capacity knob - say the width of every layer. For each width, train to convergence and record final test error. The classical expectation is a single-minimum curve: too little capacity underfits, too much overfits. Double descent is the observation that for modern over-parameterized networks the curve has **three** segments: a first descent, a rise to a peak, and then a **second descent** that can take test error below anything achieved on the classical side. A concrete version of the experiment: a 50,000-image, 10-class benchmark in which 15% of the training labels have been deliberately corrupted. Sweep width from tiny to very large. Test error improves, then gets clearly worse as width approaches a particular value, then improves again - monotonically - for every width beyond it. The widest model is the best model on the sweep, and it is better than the best model on the classical side. ## The interpolation threshold The peak is not at an arbitrary place. It sits at the **interpolation threshold**: the smallest capacity at which the model, trained with this optimizer for this budget, can drive *training* error to zero - fitting every training label exactly, including the 15% that are wrong. Two things about this definition matter in an interview. First, it is defined by what the model can *fit*, not by what it *should* fit. Interpolating a noisy training set is not a good thing; it is simply the event that marks the location of the peak. Second, for simple model families it lands roughly where the number of trainable parameters matches the number of training examples, but for deep networks that count is a poor proxy. The honest working definition is empirical: train models of increasing size and find the first one whose training error hits (essentially) zero. That point moves when you change the dataset size, the label quality, the architecture, the optimizer or the length of the schedule. ## Why the peak is there Just at the threshold, the model has *exactly* enough freedom to interpolate and none to spare. The interpolating solution is close to unique, and it is forced to pass through every corrupted label. To do that while remaining a smooth-ish function elsewhere is impossible, so the fitted function develops enormous swings around each mislabelled point. Small changes in the training sample produce completely different fitted functions - the estimator is extremely unstable - and test error is correspondingly bad. ## Why the second descent happens Past the threshold, the set of parameter settings that interpolate the training data stops being a point and becomes a large space. Every one of them has zero training error, so the training objective no longer distinguishes them. What breaks the tie is the **implicit bias** of the training procedure: gradient descent started near zero tends to land on solutions of small norm - the flattest, least contorted way of threading the same points. The bigger the model, the richer the set of interpolating functions, and the smoother the one the optimizer can find. Test error therefore falls again even though training error has been pinned at zero the whole time. This is why the phenomenon is real rather than an artifact: nothing about the training objective changed, only the size of the solution set the optimizer gets to choose from. ## The role of label noise Label noise is the ingredient that makes the bump visible. Interpolation means fitting the wrong labels too, and the cost of doing that with barely enough capacity is exactly the instability that creates the peak. Remove the corruption and the same sweep typically shows a mild bump or none at all. Any account of double descent that never mentions how clean the labels are is incomplete. The same is true of regularization: the sweeps that display a dramatic peak are usually run with weak or no explicit regularization. Tuning the regularization strength separately at each capacity can flatten the peak substantially, leaving a curve much closer to monotone. ## What this does not mean It does not mean overfitting has been repealed, that bigger is always better, or that you can read your own model's regime off somebody else's plot. It means capacity sweeps on over-parameterized models can be non-monotone, the worst place to sit is right on the threshold, and where that threshold is for *your* data is an empirical question.