What is model capacity, and why do underfitting and overfitting sit at its two ends?

## What capacity actually is A learning algorithm does not consider every conceivable function. It searches a **hypothesis space** - the set of functions the model class can express once its parameters are fitted. Linear regression on one feature can express every straight line and nothing else. A degree-15 polynomial can express every straight line *and* every wiggly curve up to degree 15. A decision tree limited to depth 3 can express at most 8 piecewise-constant regions; a tree allowed depth 30 can carve the input space into far more. **Capacity** is the size or richness of that space. It is the single most useful dial in supervised learning because almost every generalization failure is a statement about it being set wrong. A subtlety worth carrying: capacity is a property of the model *class*, not of the fitted model and not of the data. Fitting a degree-15 polynomial to 60 listings does not make the class smaller; you have merely picked one member of a very large family. This is why capacity is normally discussed before you look at any result. ## The two ends of the dial **Underfitting (the low-capacity end).** The hypothesis space simply does not contain a function close to the true relationship. No amount of optimisation helps, because the best available member is still a poor approximation. The signature is that training error is high *and* test error is high, and the two are close to each other. A depth-3 tree on 5,000 rental listings gives you eight rent predictions for the entire market; if rent genuinely depends on size, district, floor and age, eight buckets cannot express it, and the model is systematically wrong in the same direction for whole groups of listings. **Overfitting (the high-capacity end).** The space is rich enough that the procedure can find a function which follows the particular sample you drew, including the part of that sample that is noise - a landlord who priced oddly, a listing with a typo in the floor area. The signature is a large gap: training error very low, test error much higher. A degree-15 polynomial fitted to 60 listings can be made to pass essentially through every point; between the points it does whatever the algebra demands, which can mean predicting negative rent for a 45-square-metre flat because two nearby listings pulled the curve apart. ## Which dials change capacity - **Polynomial degree** - the classic textbook dial: degree 1 versus degree 15 on the same 60 listings. - **Tree depth and number of leaves** - a depth-30 tree versus a depth-3 stump on the same 5,000 listings. - **The number of input features** - each added feature enlarges the space of expressible functions. - **The number of free parameters in general** - useful as a rough intuition, though it is not a reliable measure; some one-parameter families can express astonishingly many labelings. ## Why more is not better The reason the high end fails is that you fit on a *sample*. Any finite sample carries structure that belongs to the sample rather than the population. A low-capacity model cannot chase that structure because it has nowhere to put it; a high-capacity model can, and optimisation will happily do it, because a training loss cannot tell signal from noise. That is why the two ends fail in qualitatively different ways: the low end is wrong in a stable, repeatable way, and the high end is wrong in a different way on every resampled training set. ## How to tell which end you are on Compare training error with held-out error and look at both the level and the gap: - both high, gap small - low-capacity end; - training low, held-out much higher - high-capacity end; - both low - you are somewhere sensible, assuming the held-out data really is held out. A caution: both errors being high is only *evidence* of underfitting. If the features carry no information about the target, or if the labels are intrinsically noisy, no capacity setting fixes it, and adding capacity in that situation just moves you to the other failure mode. ## Effective capacity What the model class *could* express and what it *does* express on a given dataset are not the same. Declaring a maximum depth of 30 on 5,000 listings does not produce a fully grown depth-30 tree - branches run out of examples long before that. The declared limit is an upper bound on capacity; what you actually realise depends on the data you fitted. People sometimes call this the effective capacity, and it is why the same setting can be reckless on a small dataset and conservative on a large one.