When two tree depths score within one standard error in CV, why pick the shallower tree?

## The rule, stated precisely Given a set of candidates ordered by complexity, each scored by k-fold cross-validation: 1. Find the best mean CV score (or lowest mean CV error). 2. Compute the standard error of that best candidate's CV estimate — the standard deviation of its fold scores divided by the square root of the number of folds. 3. Among **all** candidates whose mean score is within one standard error of the best (equivalently, whose error is at or below `min_error + 1 SE`), select the **simplest**. Note the direction carefully: with an error metric you move *up* one SE from the minimum and take the simplest candidate under that ceiling; with a score metric you move *down* one SE from the maximum and take the simplest candidate above that floor. Getting this backwards selects the most complex tied model, which is the opposite of the rule's intent. ## Why a tie is not a win Suppose depth 4 scores 0.808 and depth 9 scores 0.812 across five folds, with a fold standard deviation of 0.06 — a standard error near 0.027. The observed 0.004 advantage is under a sixth of the estimate's own noise. Re-run the whole thing with a different fold split and the ordering will often flip. Declaring depth 9 the winner is reading a coin flip as a measurement. Once the two are tied on the evidence, the choice must be made on something else — and complexity is the natural tiebreaker, for three reasons. **Variance.** A deeper tree fits finer structure, much of which is sample-specific. Two models with the same expected accuracy on this data distribution are not equally safe: the higher-capacity one has the wider spread of possible fitted models and typically the worse behaviour when the deployment distribution drifts a little from the training one. **Selection bias.** The maximum of the CV curve is, by construction, the point most flattered by noise — the same winner's-curse logic that inflates the best of any large candidate set. Deliberately stepping off the argmax and toward the simple end trades a fraction of an SE of apparent performance for an estimate that is less a product of the fold split. **Cost.** Shallower trees are faster to train and serve, easier to explain to a reviewer or a regulator, and less prone to surprising behaviour on rare inputs. If accuracy is tied, these are free wins. ## Where the rule is weak - **It needs a complexity ordering.** "Simplest" is only defined if the candidates line up on a single axis — depth, number of leaves, number of features, number of components. Across heterogeneous families (a shallow tree versus a linear model versus a boosted ensemble) there is no canonical ordering, so you must state the axis you are using — parameter count, inference latency, explainability — or the rule does not apply. - **One SE is a convention, not a derivation.** There is nothing sacred about one; it is a pragmatic amount of slack chosen because it is roughly the resolution of the measurement. - **The SE itself is understated.** For k-fold, the fold scores are correlated because the training sets overlap, so `s / sqrt(k)` is a floor on the real uncertainty. That cuts both ways: the true tie region is wider than one nominal SE, which is an argument for the rule's spirit rather than against it. - **Flat curves let it slide too far.** If the CV curve is nearly flat over a wide complexity range, one SE of slack can carry you deep into underfitting territory. Always look at the curve; if the simplest candidate within the band is drastically simpler than the best, sanity-check its absolute performance rather than accepting it mechanically. - **Sometimes small gains are worth money.** If a fraction of a point maps to real revenue or real harm avoided, and the complex model is affordable to run and maintain, the rule's implicit preference for simplicity is the wrong exchange rate. Say so explicitly rather than following the heuristic off a cliff. ## What it does and does not give you It gives you a defensible, reproducible tiebreak that biases toward robustness and away from the noisiest point on the curve. It does **not** give you an unbiased estimate of the chosen model's performance — the score of the model you selected is still a selection statistic, and if you need to quote a number, it has to come from data that took no part in the selection. ## The interview answer in one breath "A difference smaller than the estimate's own standard error is not a difference. I take the simplest candidate inside the one-SE band because tied models should be separated on variance and cost, not on noise — and because the exact argmax is the most noise-inflated point on the curve. It is a heuristic: it needs a complexity axis, and I'd override it if the gain, however small, were worth real money."