Why do non-parametric models often lose to simple parametric ones on a 250-row dataset?

## Flexibility is bought with rows A parametric model commits to a functional form and estimates a fixed number of quantities. A non-parametric model refuses that commitment and lets its complexity grow with the data. The refusal is not free: whatever the assumed form would have told you, the data now has to tell you instead. On a 250-row pilot the data has very little to say. ## The mechanism, in bias-variance terms Prediction error on unseen data decomposes into three parts: **bias** (error from the model family being unable to represent the truth), **variance** (how much the fitted model moves when you resample the training set), and **irreducible noise**. - The parametric model has *high bias*: if the truth is curved and you fit a plane, that gap never closes. But its variance is low, because 250 rows is a lot of evidence for a dozen coefficients — each coefficient is an average over the whole sample. - The non-parametric model has *low bias*: given enough data it could trace the true shape. But at 250 rows each of its many local decisions rests on a tiny slice of the sample. Resample the 250 rows and those decisions change. That is variance, and at small `n` variance is usually the dominant term. So the ranking flips with sample size. On learning curves — validation error plotted against training-set size — the flexible model typically starts *above* the simple one and crosses below it once there is enough data. The question is not which family is better in the abstract; it is which side of the crossover point your 250 rows sit on. ## What "nothing to interpolate between" means concretely Think of a data-growing model as building its answer out of local pieces. A neighbour-based prediction for a new market's customer is an average of the handful of stored customers closest to them. A region-based model's prediction is the average target inside the region that new point falls into. With 250 rows spread over the input space, those handfuls are small and the regions are wide, and both estimates carry large standard errors. Adding features makes it worse, because the same 250 rows must now cover a larger space. Meanwhile the parametric fit uses **all 250 rows to estimate every coefficient**. That pooling is exactly the stabilising effect you want when data is scarce, and it is why the assumed form behaves like a prior: it lets a region with no observations borrow strength from regions that have some. ## Small samples also break your ability to tell who won This is the part candidates skip. With 250 rows, a 20% holdout is 50 rows. A difference of two or three correct predictions moves the measured accuracy by several points, so the flexible model can "win" on a single split purely by luck — and if you then pick it because it won, you have selected on noise. Sound practice at this size: - **Repeated k-fold cross-validation** rather than a single split: every row is used for validation, and repeating with different splits gives you a spread, not a point. - **Report the spread.** If the two candidates' cross-validated scores overlap heavily across folds, you do not have evidence of a difference — and the tie should be broken by simplicity, stability, and how well you can explain the model to the market team. - **Keep hyperparameter search modest.** Every extra configuration you compare on the same 250 rows is another chance for noise to hand you a winner, and the reported score of the winner is optimistically biased. - **Plot a learning curve.** Fit on 50, 100, 150, 200 rows and look at where validation error is heading. If it is still falling steeply, the binding constraint is data, not model family, and the honest recommendation is to collect more. ## What this does *not* say It does not say non-parametric methods are bad, or that you should always start simple and stop. It says the flexible family's advantage is contingent on data volume, and a pilot dataset is the regime where the contingency bites hardest. It also does not say the parametric model is unbiased — it is knowingly biased, and you should say so out loud, because when the market matures and the data arrives, revisiting the choice is the plan, not an admission of error. One more nuance worth having ready: at 250 rows the usual operational arguments about a data-growing model — its storage footprint, its per-query cost — are irrelevant, because 250 rows cost nothing to keep or scan. The case against it here is purely statistical, and saying that explicitly shows you know which argument you are making. ## How to answer State the tradeoff (weak assumptions must be paid for in data), give the mechanism (local estimates from a few points, so variance dominates at small `n`), note that the ranking reverses as data grows, and finish with the evaluation caveat: on 250 rows, choose with repeated cross-validation and treat overlapping scores as a tie.