Why can a network with far more parameters than training examples still generalize?

## The paradox as an interviewer states it A network with 25 million parameters is trained on 50,000 labelled chest radiographs. It has five hundred parameters per example — enough, on a naive count, to memorize the dataset many times over. Classical intuition says this must overfit catastrophically. In practice such models routinely reach useful held-out accuracy. Explaining that without hand-waving is the question. ## What the parameter count actually measures The parameter count bounds the **size of the hypothesis class**: the set of functions the architecture can realise as its weights vary. It says nothing about which member of that class training returns. Generalization is a property of the *returned function* and of the *procedure that returned it*, not of the class alone. Confusing the two is the core error this question probes. ## The experiment that forces the issue Take a 50,000-image training set and replace every label with a uniformly random one, destroying any relationship between image and label. Train the same architecture. It reaches essentially zero training error — it memorizes the noise. Two conclusions follow, and both matter: 1. The hypothesis class really is big enough to fit arbitrary labels on the whole training set. So any capacity bound that depends only on "can this class shatter the sample" is **vacuous**: it predicts no generalization guarantee at all, for the real-label run as much as the noise run. 2. Yet the same architecture, same optimizer and same hyperparameters, trained on the *real* labels, generalizes. Since the architecture and the class did not change, the explanation cannot live in the class. It has to live in the interaction between the data, the architecture and the optimizer. A useful side observation from the same setup: random labels take substantially longer to fit than real ones. Structure in the data is *easier* to fit than noise, which tells you the training trajectory is not a neutral search over the class. ## Where the constraint actually comes from **Implicit bias of the optimizer.** Gradient descent started from small random weights does not sample the space of zero-training-error solutions uniformly. It moves in the directions the data pushes hardest, which tends to produce solutions of small parameter norm and large margin, and to fit coarse, broadly consistent structure before fitting individual examples. Among the enormous set of functions that interpolate the training set, it lands in a heavily biased corner. **Architectural priors.** Weight sharing, locality, pooling and the layer structure itself encode assumptions about the data. A convolutional trunk cannot express an arbitrary pixel-to-label lookup as easily as it expresses a translation-tolerant feature hierarchy, so the "easy" solutions it drifts toward are already the plausible ones. The effective hypothesis class the optimizer explores is far smaller than the nominal one. **Data-dependent effective capacity.** Capacity measures that actually track generalization for these models are not parameter counts — they scale with quantities like weight norms and classification margins, which the trained network controls, and which come out very differently for a network fitted to signal versus one fitted to noise. **Explicit regularization helps, but is not the explanation.** Weight decay, augmentation and early stopping improve results, and strip them all away and these models still generalize far better than a parameter count would predict. They are a tuning knob on top of the effect, not the cause of it. ## What this does not license you to say It does not mean over-parameterization is safe. It does not mean you can skip a held-out set: the argument explains an observed phenomenon, it does not certify any particular model. And it does not mean more parameters are always better — with 500 parameters per example, label noise, leakage between train and evaluation splits, and distribution shift are all still live risks, and they will bite you long before capacity does. ## How to answer well Lead with the distinction between the class and the selected function. Cite the random-label result as the reason parameter counting is the wrong instrument. Attribute the effect to the implicit bias of the optimizer plus architectural priors plus the structure of real data. Then close like an engineer: the reason to believe *this* model generalizes is a clean held-out estimate, not a theory.