skip to content

Why does a degree-15 polynomial fitted to 20 observations swing wildly between the data points?

level: middleimportance: must knowfreq 58%

answer

  1. count the free coefficients against n
  2. sixteen knobs, twenty noisy points
  3. the basis is global, not local
  4. huge coefficients that nearly cancel
  5. piecewise cubics joined at knots instead

basics

~20 s

Sixteen free coefficients against twenty noisy points leave the fit almost enough freedom to pass through every observation, so it chases noise. Polynomial columns are global and highly correlated, so small data changes swing the curve enormously.

solid answer

~50 s

A degree-15 polynomial has sixteen coefficients, and with only twenty points there is nearly enough freedom to interpolate every one of them — including their noise. Because each basis function is global, a coefficient tuned to pull the curve through one point reshapes it everywhere else, and the huge, mutually cancelling coefficients that result are extremely sensitive to the data: drop one observation and the curve changes shape completely. The oscillation is worst near the ends of the range, which is the classic Runge behaviour of high-degree fits on evenly spaced points. What I would use instead is a piecewise basis — a cubic spline with a few knots. Electricity demand against outdoor temperature is U-shaped, and three or four knots let the fit bend where it must while each basis function has local support, so a cold-weather point cannot distort the summer part of the curve.

go deeper

for a junior

Be able to say that a very high-degree polynomial can bend enough to chase individual observations, and that a curve threading every point is a warning sign rather than a success.

for a middle

Explain the mechanics: sixteen coefficients against twenty noisy points, globally supported and nearly collinear columns, and the large mutually cancelling coefficients that make the curve swing.

for a senior

Show what you would actually ship — a low-degree term where the domain says curvature exists, or a piecewise spline with a handful of knots — and demonstrate the instability first by refitting on resampled data.

for a principal

Own the standard: how much hand-specified flexibility a linear model may carry before the shape should be learned by a different model class, and how that limit is documented and enforced in review.

## Count the freedom first A degree-`d` polynomial basis contributes `d+1` coefficients: an intercept, a linear term, and every power up to `x^d`. Degree 15 therefore spends 16 parameters. With 20 observations, the fit has 16 knobs to place 20 points, leaving only 4 residual degrees of freedom. A curve with that much freedom can bend enough to pass very close to each observation, and least squares will happily do so, because the objective it minimises is training error and nothing else. The problem is that each observation is signal plus noise. A fit that reproduces the observations reproduces the noise, and the noise will be different next quarter. So the wiggles between points are not an artefact of the drawing — they are the model's actual predictions there, and they are close to arbitrary. ## Why the wiggles are so violent Two properties of the polynomial basis make the failure dramatic rather than merely mild. **The basis functions are global.** The column `x^12` is enormous at the right-hand end of the range and tiny in the middle. There is no way to adjust the fit near one observation without changing it everywhere. The fitted curve is a single tug-of-war across the whole range, and the winning configuration is typically a set of very large coefficients that nearly cancel — one term of size ten million against another of size minus ten million. Tiny changes in the data change the cancellation, and the visible curve moves a long way. **The columns are nearly collinear.** Over a fixed range, `x^11` and `x^12` trace almost the same shape, so the design matrix is badly conditioned. The coefficient estimates then have enormous variance even when the fitted values look stable. Using an orthogonalised polynomial basis fixes the *numerical* half of this — the solve becomes well behaved — but it does not fix the statistical half: the fitted function still has 16 parameters' worth of freedom and still chases noise. The end-of-range behaviour has a name. Fitting a high-degree polynomial through evenly spaced points produces oscillations that grow toward the edges of the interval — Runge's phenomenon — and it gets worse, not better, as the degree rises. ## The diagnosis you can run The convincing test is stability under resampling. Refit the same degree on the data with two points removed, or on a bootstrap resample, and overlay the curves. A degree-2 or degree-3 fit will produce a tight bundle. The degree-15 fit will produce a fan of wildly different shapes, which is exactly what high variance means: the fitted function is a property of the particular sample as much as of the underlying relationship. ## What to do instead: a piecewise basis The fix is not simply 'use degree 2 and hope'. It is to buy flexibility in a form that does not detonate. A **cubic spline** is a piecewise cubic: the range is divided at chosen points called **knots**, a separate cubic is fitted on each segment, and the segments are forced to join with matching value, first derivative and second derivative so the joins are invisible. The key property is **local support** — each spline basis function is nonzero only on a few adjacent segments. Adjusting the curve near one knot leaves the rest alone, so there is no global tug-of-war and no exploding cancellation. Electricity demand against outdoor temperature is the natural picture: demand is high in the cold from heating, falls to a minimum in mild weather, and rises again in the heat from cooling. That U shape needs perhaps three or four knots placed across the temperature range. A degree-3 spline with four knots and a degree-15 polynomial spend a comparable number of parameters, but the spline spends them locally and produces a smooth, plausible curve, while the polynomial spends them globally and oscillates. The other lever is a penalty on the coefficients, which lets you keep a rich basis while shrinking the extreme, mutually cancelling values that drive the wiggles. ## The honest limit High degree is not inherently wrong. If the relationship genuinely has that much curvature and you have thousands of observations, a degree-5 or degree-7 term can be sound. The danger is degree comparable to the number of observations, and the tell is a fit that passes through nearly every point: near-zero training error on a small sample is evidence of freedom, not of quality.

  • How does a spline basis add flexibility without the wild oscillation?
    It is piecewise: low-degree polynomials on segments divided at knots, joined so that value and the first two derivatives match. Each basis function is nonzero only near its own knots, so adjusting the curve in one region leaves the others untouched. Flexibility is bought locally, where the data says the shape bends, instead of by raising a single global degree.
  • If the fit passes through nearly every training point, why is that bad?
    Because training error then measures nothing. Each observation is signal plus noise, so a curve that reproduces the observations has memorised noise that will not repeat. The practical demonstration is refitting on a resample: the high-degree curve changes shape completely while a low-degree one barely moves, which is what high variance looks like.
  • Does using an orthogonalised polynomial basis solve the problem?
    Only the numerical half. Orthogonalising removes the near-collinearity, so the solve is stable and the coefficients become individually readable. But the fitted function still has the same number of parameters and the same freedom to chase noise, so the wiggles and the instability under resampling remain.
  • Is a high degree ever the right answer?
    Yes, when the underlying shape genuinely has that much curvature and the sample is large relative to the degree — a degree-5 term on tens of thousands of rows is unremarkable. The failure mode is degree approaching the number of observations, where the fit has enough freedom to interpolate the sample.

It is like bending a single long steel rod to touch twenty pins: forcing it onto one pin whips the rest of the rod around. A spline is several short rods hinged together, so each one only has to reach its own pins.

saying these in an interview costs you the question

  • Says more terms always fit better so must be better
  • Blames the oscillation on outliers rather than model freedom
  • Treats near-zero training error as proof of quality
  • Thinks an orthogonal polynomial basis removes the variance
  • Cannot name any alternative to raising the degree

context

resolving links…