skip to content

In k-NN, why must features be standardised before any distance is computed?

level: juniorimportance: must knowfreq 80%

answer

  1. the metric knows nothing about units
  2. widest-range column wins by default
  3. squared gaps add across columns
  4. one standard deviation as shared currency

basics

~20 s

Distance adds up per-feature gaps, so a feature measured in large units, like income in dollars, swamps one measured in small units, like age in years. Standardising puts every feature on a comparable scale so each can contribute.

solid answer

~40 s

Euclidean distance is `sqrt(sum_j (x_j - y_j)^2)`: every feature contributes its own gap, in its own units. With income in dollars beside age in years, a 10,000-dollar gap contributes 10,000 while a 30-year gap contributes 30, so the squared sum is decided by income alone and age is effectively ignored — the neighbours you retrieve are income neighbours. Worse, the model is not unit-invariant: expressing income in thousands instead of dollars changes which points are nearest and therefore changes the prediction. Standardising each feature — subtract its mean, divide by its standard deviation — makes one standard deviation of movement cost the same in every feature, so the metric measures what you intended. The same argument holds for Manhattan and Minkowski distance, and cosine distance is not exempt either.

code

python · 21 lines
python
import math

def dist(a, b):
    return math.sqrt(sum((x - y) ** 2 for x, y in zip(a, b)))

# (annual income in dollars, age in years)
query = (52000, 30)
cand_a = (52500, 62)   # nearly the same income, 32 years older
cand_b = (61000, 31)   # nearly the same age, 9k more income

print(round(dist(query, cand_a), 1), round(dist(query, cand_b), 1))
# 501.0 9000.0  -> A is 'nearest', on income alone

means = (55000, 40)
sds = (15000, 12)          # standardise: (value - mean) / sd

def z(point):
    return tuple((v - m) / s for v, m, s in zip(point, means, sds))

print(round(dist(z(query), z(cand_a)), 2), round(dist(z(query), z(cand_b)), 2))
# 2.67 0.61  -> B is nearest, which is what the data actually says

go deeper

for a junior

Be ready to say, in one breath, that distance sums per-feature gaps in their raw units, so a big-unit feature like income decides every neighbour until you standardise. Name a concrete pair of features when you say it.

for a middle

Explain the mechanics: which term dominates the squared sum, that the model is not invariant to a change of unit, and that Manhattan and Minkowski have the same weakness while Hamming and Gower do not.

for a senior

Show you would catch this in a real pipeline — neighbour lists that look sensible on one column and random on the rest, and a diagnosis by inspecting per-feature contributions to the distance rather than only the total.

for a principal

Own the framing that standardisation is an assertion of equal importance per standard deviation, not neutral hygiene. Decide when explicit feature weights or feature removal is the honest expression of domain knowledge instead.

## What a distance actually computes A nearest-neighbour model has no learned weights. Its entire notion of "similar" is the distance function, and the most common one is Euclidean distance between two records `x` and `y` with `d` features: ``` d(x, y) = sqrt( (x_1 - y_1)^2 + (x_2 - y_2)^2 + ... + (x_d - y_d)^2 ) ``` Read that sum carefully: each feature contributes the square of its own raw gap, **in whatever units that feature happens to be recorded in**. The formula contains nothing that normalises those units. Manhattan distance, `sum_j |x_j - y_j|`, has exactly the same property. ## The worked failure Take a two-feature customer table: annual income in dollars (roughly 20,000 to 200,000) and age in years (roughly 20 to 80). Compare a query customer earning 52,000 at age 30 against two candidates: - Candidate A earns 52,500 and is 62 years old. Gaps: 500 in income, 32 in age. Squared sum: 250,000 + 1,024, so distance about 501. - Candidate B earns 61,000 and is 31 years old. Gaps: 9,000 in income, 1 in age. Squared sum: 81,000,000 + 1, so distance about 9,000. Candidate A wins by a factor of eighteen, and A is a person thirty-two years older than the query. Age contributed 1,024 out of 251,024 — about 0.4% — of A's squared distance, and essentially nothing of B's. The age column is present in the data and absent from the model. Nobody gets an error; the model trains, scores and ships, and it is a one-feature model wearing a two-feature costume. Now divide income by 15,000 and age by 12 (roughly their standard deviations). A's gaps become 0.03 and 2.67, giving distance 2.67. B's become 0.60 and 0.08, giving distance 0.61. The nearest neighbour flips to B — the person of the same age and similar income, which is what a human would have said all along. ## Unit-invariance is the sharp way to see it A decision tree splits one feature at a time on a threshold, so rescaling a feature monotonically leaves every split — and every prediction — untouched. A distance-based model has no such invariance: rewriting income in thousands rather than dollars divides its contribution by a million and can reorder every neighbour list. Any model whose behaviour depends on an arbitrary choice of unit is under-specified until you pin the units down, and standardising is how you pin them down. ## What standardising actually asserts Dividing every feature by its standard deviation is a **claim**, not a neutral cleanup: it says that one standard deviation of movement in any feature is worth the same as one standard deviation in any other. That is a far better default than "whatever unit the source system used", but it is still a default. If domain knowledge says one feature matters more, express that explicitly with feature weights inside the metric, `d = sqrt(sum_j w_j (x_j - y_j)^2)`, and tune the weights the same way you tune k. Setting a weight to zero drops the feature entirely. The corollary bites too: after standardisation, a pure-noise column is weighted exactly as heavily as your best predictor. Distance-based models therefore degrade quickly when you throw in irrelevant features, and feature selection matters more here than for models that can learn to ignore a column. ## Which metrics are affected All of the summed-per-feature metrics are: Euclidean, Manhattan and any Minkowski distance. Cosine distance is a common false exemption — it removes the *magnitude of the whole vector*, not the mismatch of units between columns, so a dollar-scale column still dominates the underlying dot product. Hamming distance over binary indicators and Gower-style dissimilarities are the exceptions, because each per-feature contribution is already constructed to live on a common 0-to-1 scale. One-hot indicator columns need thought rather than a reflex: standardising a rare category's dummy divides it by a very small standard deviation, which can make a single mismatch on a rare level outweigh everything else. ## The boundary of this answer Which transform you use — z-score, min-max, or an outlier-resistant variant — and the discipline of computing its statistics from training data only are preprocessing concerns. The point that belongs to the metric itself is narrower and non-negotiable: without *some* common scale, the distance is not measuring what you think it measures, and every downstream choice of k, of weighting, or of search structure is being tuned on top of a broken ruler.

  • Does a decision tree need the same treatment before it splits?
    No. A tree tests one feature against a threshold, so any monotone rescaling of that feature moves the threshold and leaves the split — and the prediction — identical. Scale sensitivity is specific to methods that combine features into one number, such as distance-based and gradient-based models. That is why a tree can be handed raw dollars and years without complaint while a neighbour model cannot.
  • If one feature genuinely matters more than the others, how do you say so after standardising?
    Put weights inside the metric: `d = sqrt(sum_j w_j (x_j - y_j)^2)` on the already-standardised features. A weight of 2 makes that feature count twice as much per standard deviation; a weight of 0 removes it. Tune the weights on validation data alongside k. This is deliberate, inspectable weighting, unlike the accidental weighting you get from leaving raw units alone.
  • Does using cosine distance remove the need to standardise?
    No. Cosine removes the length of the whole vector, not the mismatch of units between columns. A dollar-scale column still contributes most of the dot product and most of each vector's norm, so it still dominates the angle. Cosine solves 'this user is five times more active', not 'this column is measured in thousands and that one in single digits'.

Ranking runners by adding their finishing time in seconds to their height in metres: the seconds dwarf the metres, so you have secretly ranked on time alone.

saying these in an interview costs you the question

  • Claims k-NN is scale-invariant the way a decision tree is
  • Says scaling is only about making the computation faster
  • Believes cosine distance makes standardisation unnecessary
  • Thinks only the target variable needs rescaling
  • Assumes standardisation also removes irrelevant features

context

resolving links…