Why does one extreme feature value distort an OLS fit and kNN neighbourhoods but barely move a decision tree?

## One row, three different fates Take a table where one row carries a feature value far outside the rest of the column. Whether that row is a problem depends entirely on what the learner does with numbers, and the three canonical answers cover most of classical ML. ## Least squares: the square is the whole story OLS chooses coefficients to minimise `sum((y_i - yhat_i)^2)`. Squaring is what makes it fragile. A row with a residual of 1 contributes 1 to the objective; a row with a residual of 10 contributes 100. The optimiser will happily worsen the fit on fifty ordinary rows if that buys a big reduction on the one enormous residual, because the arithmetic rewards it. The direction of the damage depends on *where* the extreme sits. - **Extreme in `y` at an average `x`.** The fitted line mostly shifts up or down; the intercept absorbs it. The row leaves a large visible residual, which makes it comparatively easy to notice. - **Extreme in `x` (high leverage).** The fit pivots roughly about the centre of the data, so a point far along the `x` axis has a long lever arm and can swing the slope. Worse, the line often chases it, so its own residual ends up small and the point hides. This is why the phrase *high leverage* exists as distinct from *large residual*. A penalised linear model is not immune either: the coefficients are still fitted on the same squared loss, and if the feature is not scaled, an extreme value also changes what the penalty means for that coefficient. ## Nearest neighbours: distance sums the gap kNN and every distance-based method (k-means, kernel methods, anything on Euclidean geometry) compute `d(a, b)^2 = sum over features of (a_f - b_f)^2`. One feature with an extreme value contributes an enormous term, and it swamps the contribution of all the other features. The consequences: - The outlier row is very far from everything, so it is nobody's neighbour, and its own `k` neighbours are essentially arbitrary rows chosen by the remaining features. - If you standardise the column afterwards, the extreme value inflates the standard deviation, so every ordinary row is compressed into a narrow band and the geometry among *them* degrades. This is the counter-intuitive part: the outlier harms the rows that are not outliers. So for distance-based learners the question is not whether the extreme row is predicted well, but whether it has quietly rescaled the space that all the other rows live in. ## Trees: only the ordering is read A decision tree evaluates candidate splits of the form `feature <= t`. The set of possible partitions of the rows depends only on the **ordering** of the feature values, not on their magnitudes. Any strictly increasing transformation of the column, including replacing 400 with 4,000,000, leaves every achievable partition identical, so the fitted tree is unchanged. The extreme row simply ends up alone or nearly alone on one side of the outermost split, contained in a small leaf, and the rest of the model is untouched. This is exactly why *scale-free* is the standard description of trees and why they need neither standardisation nor feature capping. ## The two caveats that separate a good answer from a great one **Extreme targets still hurt trees.** A regression tree chooses splits by reducing squared error in `y` and predicts the mean of `y` in each leaf. Both are non-robust. One enormous target value will attract splits that exist only to isolate it, and it will inflate the prediction of whatever leaf it lands in. Feature outliers are harmless to trees; target outliers are not. **Gradient boosting inherits the loss.** The tree *structure* is order-invariant, but boosting fits successive trees to the gradients of a chosen loss. Under squared error those gradients are the residuals, so a single extreme target dominates the early rounds exactly as it dominates an OLS fit. Naming XGBoost, LightGBM or CatBoost does not change this; it is a property of the objective, not the implementation. ## What to do with the finding The practical rule is that outlier treatment is not a universal cleaning step but a model-family decision made together with the choice of learner: - Distance-based or linear model on a heavy-tailed feature: treatment usually earns its keep, whether that means capping the feature or choosing a scaler that the tail cannot dominate. - Tree ensemble on the same feature: usually leave it alone, but inspect the **target** distribution and be aware that a live value beyond the training range falls into the outermost leaf, where the prediction is flat. And whichever route you take, the treatment is part of the model pipeline, applied identically at training and at serving.