When would you use median/IQR robust scaling instead of z-score standardisation?

## The failure it fixes Standardisation divides by the standard deviation, which is computed from squared deviations around the mean. Squaring means the largest deviations dominate the sum, so one or two extreme rows can raise the standard deviation far above what the typical spread of the column looks like. When that happens, the standardised column has a peculiar shape: the vast majority of rows sit in a tiny interval such as -0.05 to 0.05, while a small number sit at 20 or 40. For a model that cares about differences between rows, that is close to having thrown the feature away. Two typical examples now differ by hundredths of a unit on this column while differing by whole units on every well-behaved column, so the feature stops contributing to the comparison even though it is perfectly informative for ordinary rows. ## The transform Robust scaling swaps both statistics for order statistics: ``` x' = (x - median) / IQR where IQR = Q3 - Q1 ``` `Q1` and `Q3` are the 25th and 75th percentiles. Substituting the two quartiles into the formula shows the geometry directly: `Q1` maps to `(Q1 - median) / IQR` and `Q3` to `(Q3 - median) / IQR`, and the two differ by exactly 1. So the middle half of the data always occupies a unit-width interval, and the median sits at 0. The key property is breakdown behaviour. The median and the quartiles are determined by the *position* of values, not their magnitude: you can push the largest 10% of the column to arbitrarily large numbers and the median will not move at all, and the quartiles barely. The mean and standard deviation have no such protection — a single value can drag either as far as you like. ## What it deliberately does not do **It does not bound the output.** Robust scaling is not min-max with a different centre. A row far above `Q3` stays far above; after scaling it might sit at 45. The scaled column is unbounded in both directions, exactly like a standardised one. **It does not treat the extremes.** Nothing is dropped, clipped or winsorised. The extreme rows are still present with their original relative position; the transform only chose a centre and a unit that ignore them. Whether those rows should be capped, removed, or kept as genuine signal is a separate decision made on the data's meaning, not something a scaler decides for you. **It does not change shape.** Like every other affine rescale, it moves and stretches without bending. A skewed column stays skewed. ## When it is the wrong choice **A zero interquartile range.** If more than half of a column's values are identical — very common for count columns that are mostly zero, or for a flag-like numeric field — then `Q1 = Q3`, `IQR = 0`, and the division is undefined. Any implementation of the idea needs a rule for that case: leave the column unscaled, drop it as near-constant, or use a wider pair of percentiles such as the 5th and 95th. Discovering this in production because a batch of scaled values came back as infinities is a bad way to learn it. **A well-behaved column.** If the column has no heavy tail, the standard deviation and the IQR carry the same information and standardisation is the more conventional, more widely understood choice. Robust scaling is not strictly better; it is a targeted answer to a specific problem. **When the extremes are the signal.** In fraud or fault detection the enormous values are often the cases you most want the model to separate. Rescaling so that the ordinary rows spread out is usually still fine — the extremes remain far away — but be sure you are not reasoning as though robust scaling somehow protects the model from them. ## Diagnosing which you need Before choosing, compare the two summaries of the same column: mean against median, and standard deviation against IQR. When the mean sits far from the median, or the standard deviation is several times the IQR, extreme values are steering the mean-based statistics and robust scaling is the safer transform. After scaling, look at the distribution of the output — if nearly every row landed inside a hair's width of zero, the divisor was set by the tail, and that is the symptom you were trying to avoid.