Algorithm Choice Heuristics

### The decision you are actually making Choosing a model family is a search-narrowing decision, not a search. On a new tabular supervised problem there are only a handful of families worth considering at all: regularized linear models, tree ensembles, distance-based methods, kernel methods, simple probabilistic classifiers, and neural networks. The useful skill is eliminating most of them in about a minute from the shape of the data and the constraints around the model, so the time you have goes into features, labels and evaluation rather than into a bake-off nobody asked for. ### Step one: the linear or logistic baseline Fit a regularized linear model (linear regression for a continuous target, logistic regression for a binary one) before anything else. It costs seconds, has almost nothing to tune, and pays for itself four ways. - It produces the number every later model must beat. Without it, a boosted model scoring 0.91 AUC is a number with no meaning; with it, you know whether 0.91 is a triumph over 0.72 or a rounding error over 0.90. - It is a leakage detector. A baseline that scores near-perfectly on a problem experts find hard almost always means a column encodes the answer, a row was duplicated across the split, or the label was computed from a feature. - It exercises the whole pipeline end to end, so when a complex model later behaves oddly you already know the splits, the metric and the feature build work. - Its coefficient signs are a cheap sanity check. A feature whose weight points the opposite way from every domain expectation is worth a look before you hide it inside an ensemble. ### Step two: gradient-boosted trees as the tabular default If the baseline is not good enough, the standard next reach on mixed-type tabular data is a gradient-boosted tree ensemble. The reasons are properties of the family, not brand loyalty. - Mixed column types are native. Numeric, ordinal and categorical columns coexist without a common scale, and a heavy categorical such as a postcode with tens of thousands of levels can be split on directly; some boosting algorithms (CatBoost and LightGBM among them) handle categorical splits inside the algorithm itself. - Splits depend only on the order of a feature's values, so scaling, standardising or taking logs of an input changes nothing. Any monotone transform of a feature leaves the fitted model identical. - Interactions and non-linear thresholds come for free. A tree of depth d can express an interaction among d features that you never wrote down, and a step at any cut point captures effects such as risk changing sharply above a certain age. - Outliers in the inputs are harmless: an extreme value is simply on the far side of a split rather than a large number multiplying a weight. - Many boosting algorithms have a defined rule for routing missing values down one branch, so a sparsely populated column does not force an imputation decision up front. What the family cannot do matters just as much. Trees do not extrapolate: outside the range of the training data the prediction is flat, so a model that must forecast beyond observed values is a bad fit. A smooth, near-linear relationship needs many splits to approximate as a staircase, which is one reason the linear baseline sometimes simply wins. ### What overrides the default The heuristic is a starting point, and four situations override it before you look at any score. - Very few rows. With a few hundred rows a boosted ensemble has ample capacity to fit noise, and you do not have enough data to tune it or to measure the difference reliably. - Far more columns than rows. A flexible family cannot be supported by the data, and a strongly regularized linear model is the honest choice. - Wide sparse data such as bag-of-n-gram text, where the signal is spread additively over tens of thousands of mostly-zero columns. - A hard external constraint: a scoring rule a person must be able to read, a strict latency or memory budget at serving time, or a model that must be updated continuously rather than refit. Constraints are filters, not tie-breakers. Apply them first and let accuracy decide only among the families that survive. ### Running the comparison honestly Shortlist two or three families, not seven. Fix the evaluation protocol and the metric before you fit anything, give each candidate a comparable tuning budget, and compare once. The common failure is not picking the wrong family; it is tuning the challenger against the same held-out data that produced the baseline's score, so the winner is decided by how many times you looked.