Bias-Variance and Generalization

A held-out score is an average of errors over a sample of rows. It estimates future error only if future rows are independent draws from that same distribution. When the population or the relationship moves, the number stops applying.

Bias is the gap between the true value and the average prediction a model makes when it is retrained on different training samples. Variance is how much that prediction moves from one training sample to the next.

Training error is optimistic because the fitting procedure tuned the model to those exact rows, absorbing their random noise as if it were signal. On fresh rows the noise does not repeat, so error rises.

A validation curve plots training score and held-out validation score against one hyperparameter that controls model capacity. Pick the value where validation peaks; a widening gap between the two lines past that point is the overfitting signal.

Enlarging a nested hypothesis space keeps every earlier fit available, so training error can only fall. Test error instead trades falling systematic error against rising sensitivity to the sample, so it bottoms out and climbs.

Covariate shift moves the input distribution p(x) while p(y|x) holds. Label shift moves the class balance p(y) while p(x|y) holds. Concept drift moves p(y|x) itself, which is the only one that makes the model wrong.

Curves that meet at a high error mean high bias: the model is too simple, and more rows will not help. A gap that stays wide at every training size means high variance, where more rows still help.

A 12-point train-validation gap is high variance, not high bias. Rank the fixes by cost: more training rows, then a stronger penalty or fewer features, then a lower-capacity model. Adding capacity would make the gap worse.

The expectation runs over two random things: the training set the model was fitted on, drawn afresh from the same population, and the noise in the new label being predicted. The input point and the fitting procedure are held fixed.

Part of the error is irreducible: the label is not fully determined by the features, so near-identical inputs carry different labels. Data cannot remove that. The rest of the floor is the model's own bias, which rows also cannot fix.

An inductive bias is the assumptions a learner uses to choose among hypotheses that fit the training data equally well. Data alone never picks one, so without a bias there is no basis for predicting a new row.

The no-free-lunch theorem says that averaged uniformly over every possible target function, all learning algorithms have the same expected error on unseen inputs. No algorithm dominates in general; an advantage only exists relative to a restricted class of problems.

Three things set the gap: the fitting procedure's effective degrees of freedom, the noise variance of the target, and the number of rows. The expected gap scales as degrees of freedom times noise, over rows.

The small-k end. At k = 1 every training point is its own nearest neighbour, so training error is zero by construction. Capacity falls as k grows, so this curve's complexity axis runs right to left.

More training data lets you afford more capacity. Sensitivity to the sample shrinks as the sample grows while a class's systematic error does not, so the best capacity moves upward with dataset size.

Vary only the training-set size. Score every point against one fixed held-out set, draw several random subsamples per size and average them, space sizes logarithmically, and refit preprocessing inside each subsample so nothing leaks.

Most of the error is avoidable bias: three points sit between the model's 9% training error and the 6% human benchmark, against a one-point train-validation gap. Attack bias first - more capacity, richer features, weaker regularization - not more data.

That disagreement estimates the irreducible-noise term of the error decomposition. No model of the recorded features can beat it on average, so it sets the floor your error should be judged against rather than against zero.

The in-sample Sharpe was never a forecast: it is the best of many rules scored on the very history that selected them. A maximum over noisy statistics is biased upward, so the winner's number was inflated from the start.

Not as a statement about 2021. The 2019 held-out AUC describes the 2019 applicant mix. Re-estimate it under the new mix where the two populations overlap, and cap exposure where 2019 gave no coverage at all.

VC dimension is the largest number of points a classifier class can label in every possible way. Some 3 non-collinear points admit all 8 labelings by a line, but no 4 points do, so the answer is 3.

Far fewer than 20,000. Reviews from one seller are correlated, so they repeat information rather than adding it. With a design effect near 20 the sample behaves like about a thousand independent rows, so every error bar widens.

Squared error is quadratic, so expanding it leaves exactly three additive non-negative terms. Counting misclassifications is a step function of the prediction error, so no such algebra exists, and instability across refits can even lower the error rate.

An edge value is a property of your grid, not of the model: the sweep ended before the optimum. Extend the range and refit until validation clearly flattens or turns over, then choose.

Extrapolate the curve one or two doublings at most, price the projected error drop in hours and money, and ask whether it changes a decision the product makes. Buy a small tranche first to test the projection.

No algorithm is best on every problem, so the claim is empirical — a statement about the problems that team has seen. Ask what evidence supports the default and which observable conditions void it.

Pairwise agreement of 92% is not an 8% error floor. If annotators err independently, each is wrong on roughly 4% of items. Adjudicate a sample to split irreducible ambiguity from fixable guideline drift before funding more modelling.