Why must every learning algorithm carry an inductive bias to generalize?

## What an inductive bias is An inductive bias is the collection of assumptions a learning algorithm uses to prefer some hypotheses over others when the training data cannot distinguish between them. It is the answer to the question: *of all the rules that fit what I have seen, why this one?* The need for it is not a practical inconvenience — it is logical. Induction from finite observations to a general rule is not valid deduction. The data constrains the hypothesis on the observed inputs and nowhere else. Any bridge from `what I saw` to `what I will see` is built out of assumptions, and the only question is whether they are stated or smuggled in. ## The sequence argument Show someone 2, 4, 6 and ask for the next term. The usual answer is 8, from the rule 'the positive even numbers'. But 'the even numbers below 7, then zero forever' also fits and predicts 0. And for *any* value v you name, there is a cubic polynomial passing through the points (1,2), (2,4), (3,6) and (4,v). The observations are equally consistent with all of these; they cannot select one. What makes 8 feel obviously right is not the data — it is a prior over rules that ranks 'add two' above 'add two, then behave arbitrarily'. That prior is doing the predicting. Every learning algorithm is in the same position on every dataset, just with more dimensions and less obvious rules. ## What the bias looks like in real model families Each family encodes a different prior, and naming it is the middle-level skill an interviewer is checking: - **Linear models**: the target is approximately a weighted sum of the features; effects are additive and monotone in each feature unless you build interactions or transforms yourself. - **k-nearest-neighbours**: points that are close under the chosen distance have similar labels — a smoothness or local-constancy assumption. Note that the *distance* is itself part of the prior: scaling a feature changes what 'close' means. - **Decision trees and their ensembles**: the target is well approximated by a piecewise-constant function over axis-aligned regions. This makes them insensitive to monotone rescaling of a feature and comfortable with sharp thresholds, but unable to extrapolate a trend beyond the range of values seen in training. - **Naive Bayes**: features are conditionally independent given the class — a very strong, usually false assumption that nonetheless often ranks classes well. None of these is 'the correct' bias. They are different bets, and the winner on a given problem is whichever bet the data happens to reward. ## Representation is where most of the prior lives Before any model runs, feature choice and encoding have already injected a prior. Watanabe's *ugly duckling theorem* makes this sharp: if you describe a set of objects by all logical predicates over their attributes, then any two distinct objects share exactly the same number of predicates. Under that fully neutral description, every pair of objects is equally similar, and 'similarity' carries no information. Similarity only becomes meaningful once you privilege some features over others — that is, once you bring a prior. So a claim like 'the model found these two customers similar' always rests on the feature set and scaling somebody chose. A vivid version: take an image dataset and permute the pixel order with one fixed permutation applied identically to every image. A learner whose bias assumes neighbouring inputs are related is badly damaged, while a learner that treats features as an unordered set is entirely unaffected — its fit is identical up to relabelling. The data content is unchanged; only the match between the prior and the representation changed. ## Strength versus correctness A strong bias is not a defect and a weak one is not a virtue. What matters is *match*. A strong, correct assumption lets you learn from very few rows, because the data only has to select among a small family of plausible rules. A strong, wrong assumption locks in a systematic error no amount of data will remove. A weak bias hedges: it can eventually represent the right rule, but it needs far more data to find it, and it is easier to fool with noise. This is also why 'more data removes the need for assumptions' is false. More data narrows which hypotheses stay consistent; it never reduces their number to one, and the leftovers still have to be ranked by something. ## Saying it well Define inductive bias, give one concrete illustration where the data underdetermines the answer, name the assumption behind a model you actually use, and close with the practical consequence: model selection is the act of choosing which assumptions to make, so you should be able to state your model's assumptions and point at evidence that the data respects them.