Why do kNN and SVM need feature scaling while decision trees do not?

## The one distinction that decides it Ask a single question about the model: **does it combine several features into one number whose value depends on their units?** If yes, it is scale-sensitive. If it examines features one at a time through comparisons, it is scale-free. ## The scale-sensitive families **Distance-based learners.** k-nearest-neighbours ranks candidate neighbours by squared Euclidean distance, a sum of per-feature squared differences. Consider a diabetes-risk model with three features: fasting glucose in mg/dL (roughly 70-200), BMI (roughly 18-45), and annual household income in dollars (roughly 20,000-200,000). A 30,000-dollar income gap contributes a squared term of nine hundred million; a clinically enormous 50 mg/dL glucose gap contributes 2,500. The glucose and BMI terms are numerically invisible. The model still trains, still returns predictions, and is silently a nearest-neighbour model on income. Standardising all three puts their contributions on the same footing. k-means shares the mechanism, since its assignment step is the same distance comparison. **Kernel methods.** An RBF-kernel SVM computes similarity as a decaying function of squared Euclidean distance between examples, so it inherits the distance problem exactly. On wine-chemistry features where proline runs in the hundreds of units and pH sits near 3, the proline differences swamp the pH differences inside the kernel. Worse, a single kernel width has to serve every feature at once — there is no per-feature knob to compensate with — so no hyperparameter search rescues an unscaled fit. Linear SVMs are affected too, because the margin is measured in the same geometry. **Variance-based decompositions.** PCA finds the directions of greatest variance, and variance is measured in the square of whatever unit the column uses. On a 20-channel semiconductor process panel measured in pascals, millivolts and degrees Celsius, the channel with the numerically largest units carries the largest variance and therefore drags the first component toward itself. Rescale the channels and the component ordering changes — the geometry was never intrinsic, it was a property of the measurement units. **Penalised linear models.** A single penalty strength is applied to every coefficient, but a coefficient's magnitude depends on its feature's units: express a length in metres instead of kilometres and its coefficient shrinks by a thousand. So the penalty falls unevenly across features unless they are first put on a common scale. **Gradient-based optimisation.** Even where scaling does not change the *solution*, features on wildly different scales make the loss surface long and narrow in some directions, so a gradient method takes many small zig-zagging steps to converge. This is a speed and stability argument rather than a correctness one. ## Why trees are exempt A decision tree grows by searching, feature by feature, for a threshold `t` and a split of the form `feature j <= t`. Which rows fall to the left child depends only on **how the values of that one feature are ordered**, not on their magnitudes and not on any other feature's units. Standardisation and min-max ranging are strictly increasing maps: they never reorder values. So the set of distinct ways to partition the rows on that column is identical before and after scaling, the class proportions in each candidate child are identical, the impurity gains are identical, and the algorithm picks the same partition. The chosen threshold is simply reported on the new scale — a split at glucose `<= 126` becomes a split at the standardised equivalent of 126. Everything built from axis-aligned trees inherits this: bagged forests, random forests, and gradient-boosted tree algorithms such as XGBoost, LightGBM and CatBoost. A frequent confusion is that gradient boosting must need scaling "because it uses gradients" — but the gradient steps happen in function space, fitting each new tree to the residual signal, while the trees themselves still split on unscaled thresholds. ## The case that catches people out An unpenalised linear or logistic regression fitted to convergence is **not** scale-sensitive in its predictions. Multiply a feature by 1,000 and its fitted coefficient divides by 1,000; the fitted function, and therefore every prediction, is unchanged. Scaling matters there for two secondary reasons: gradient-based fitting converges faster on a well-conditioned problem, and coefficients on a common scale can be compared to each other, which raw-unit coefficients cannot. ## How to answer in an interview Do not recite a list. State the rule — geometry versus per-feature comparisons — then derive the list from it, and be explicit that scaling is not a universal ritual: on a tabular problem fed to a gradient-boosted tree, it buys nothing at all.