Scaling and Standardization

## Why rescale at all A table of numeric features usually arrives in whatever units the source system happened to use: a glucose reading in mg/dL, a body-mass index around 25, a household income in dollars. Several model families compare or combine feature values directly, and when they do, the column with the biggest numbers wins by accident of measurement rather than by relevance. Scaling is the family of transforms that removes that accident. ## Z-score standardisation The formula is ``` z = (x - mean) / sd ``` where `mean` and `sd` are the column's mean and standard deviation, computed once on the training data and then reused unchanged for validation, test and production rows. After the transform the column has mean 0 and standard deviation 1. The output is unbounded: a value five standard deviations above the mean becomes 5.0, and nothing clips it. Both statistics are computed from every row, so every row has some influence on the result. That is usually what you want, and it is also the weakness: the mean and especially the standard deviation are pulled by extreme values, so one enormous observation can inflate `sd` and squeeze the rest of the column into a narrow band around zero. ## Min-max ranging The formula is ``` x' = (x - min) / (max - min) ``` using the training minimum and maximum. Training values are mapped onto `[0, 1]`; the minimum becomes exactly 0 and the maximum exactly 1. Any target interval works — subtracting and rescaling again gives `[-1, 1]` — but the character is the same: the mapping is determined entirely by two order statistics, the two most extreme rows in the column. Every other row is only positioned relative to them. Two consequences follow. First, a single freak maximum decides the whole mapping, and every ordinary value gets pushed toward the bottom of the interval. Second, the bounded-output guarantee only holds for data inside the training range. A new row above the training maximum maps above 1, and a new row below the training minimum maps below 0. Nothing breaks numerically, but if a downstream component genuinely requires inputs in `[0, 1]`, you have to decide whether to clip — and clipping throws away the distinction between merely large and enormous. ## Max-abs scaling A third linear option divides by the largest absolute value in the column: ``` x' = x / max(|x|) ``` Values land in `[-1, 1]`, and because nothing is subtracted, an exact zero maps to an exact zero. That matters for wide, mostly-zero count columns: subtracting a mean would turn every stored zero into a non-zero number and destroy the sparsity that made the representation cheap in the first place. ## What none of them do All three are affine transforms — multiply by a constant, add a constant. Affine transforms move and stretch the number line; they do not bend it. So: - Standardising does **not** make a column normally distributed. A right-skewed column standardised to mean 0 and sd 1 is exactly as skewed afterwards. - Scaling does not remove outliers. An extreme row is still extreme, just expressed in new units. - The Pearson correlation between two columns is unchanged by rescaling either of them (with a positive multiplier), and so is the ordering of values within a column. Changing the *shape* of a distribution requires a nonlinear transform, which is a different tool for a different problem. ## Choosing between them Standardisation is the sensible default for the scale-sensitive families — distance-based learners, kernel methods, variance-based decompositions and penalised linear models — because those methods care about spread, and standardisation is what equalises spread. Min-max earns its place when the column is genuinely bounded and you know the bounds from the domain rather than from the sample: a 0-to-10 satisfaction score, a percentage, a rating with a fixed ceiling. Then `min` and `max` are constants of the problem, not sample accidents, and out-of-range values cannot appear. It is also the right answer when a downstream component requires bounded inputs. Max-abs is the sparse-data choice, for the zero-preservation reason above. ## A note on vocabulary "Normalisation" is used loosely in practice: some people mean min-max ranging, some mean z-score standardisation, some mean scaling a row vector to unit length. In an interview, say which formula you mean rather than relying on the word — and if the interviewer uses it, ask.