Squared-Error Regression

## The word 'linear' describes the parameters A linear regression model has the form `y = b0 + b1*z1 + b2*z2 + ... + bm*zm + error` where `z1 ... zm` are columns of known numbers computed from the data and `b0 ... bm` are the coefficients to be estimated. The word *linear* describes how the coefficients enter: each multiplies one column, and the pieces are added together. It says nothing about how those columns relate to the raw measurement they were built from. So take a single measured input `x` and construct two columns, `z1 = x` and `z2 = x^2`. The model `y = b0 + b1*x + b2*x^2` is a linear model in exactly the technical sense that matters, because once `x^2` has been computed it is just a number sitting in a column. The fitted relationship between `x` and `y` is a parabola — visibly not a straight line — and there is no contradiction. ## The design-matrix view The practical way to see it is to picture the design matrix: one row per observation, one column per term, with a leading column of ones for the intercept. For an observation with `x = 3`, the row of a quadratic model is `[1, 3, 9]`. Squaring happened before the model saw anything. Least squares then chooses the coefficients that minimise the sum of squared residuals, and that minimisation problem has exactly the same structure — a quadratic in the coefficients with a single minimum — whether the columns hold raw measurements, squares, cubes, products of two features, or spline basis functions. This is why the technique is called **basis expansion**. You replace the single input `x` with a set of basis functions evaluated at `x`, say `f1(x), f2(x), ..., fm(x)`, and fit a linear combination of them. Polynomial terms are one choice of basis; piecewise-cubic spline basis functions defined by knots are another; the product of two features is another. The model class stays linear-in-parameters throughout, and everything that follows from that — a convex objective, a unique solution for a full-rank design, closed-form standard errors — is preserved. ## Why anyone bothers Crop yield against nitrogen dose is the standard picture. Yield climbs steeply with the first units of nitrogen, the gain flattens, and past some dose yield actually falls as the excess damages the crop. A model with only a nitrogen column can fit one slope, so it will report either a positive or a negative average effect and be wrong over most of the range. Adding a squared nitrogen column gives the fit the freedom to rise and then turn over, while every property of the fitting procedure is unchanged. ## What nonlinear in the parameters looks like Contrast `y = b0 * exp(b1*x)`. Here `b1` is inside an exponential, so the objective as a function of `b1` is no longer a simple quadratic. There is no single linear solve that returns the answer; you need an iterative optimiser, sensible starting values, and a check that it converged to a good optimum rather than a poor local one. The same is true of a logistic growth curve with an unknown rate and asymptote. The distinction is not academic: it decides whether the fit is a deterministic one-shot computation or a numerical search. ## The costs that come with the extra columns Two caveats belong with the answer, and mentioning them separates a memorised definition from understanding. First, the added columns are usually strongly correlated with the ones they came from. Over a range of positive doses, `x` and `x^2` move together almost in lockstep, which makes the design matrix poorly conditioned: individual coefficients become unstable and hard to read even though the fitted curve is fine. Rescaling the input, or using an orthogonalised polynomial basis, addresses the numerical side of this. Second, every column added is another parameter estimated from the same data. Flexibility bought this way is paid for in the stability of the fit, and the payment grows quickly with the degree. ## The one-line answer Linear regression is linear in its coefficients. Squares, cubes, splines and interaction products are all just columns you compute first, so the model remains a linear model while the relationship it expresses can be as curved as the basis allows.