A k-NN uses 6 numeric columns and 5,000 one-hot merchant-ID columns and predicts poorly — how do you fix it?

## Diagnose before you fix Three measurements settle this in minutes. 1. **Contrast ratio.** For a sample of queries, compute `(farthest - nearest) / nearest` against the training rows. If it is small — well under one — the metric has lost its ability to rank. 2. **Neighbour composition.** For each query, count how many of its `k` neighbours share its merchant ID. If that number is far above what the merchant's frequency would produce by chance, the ID is driving the neighbourhood. 3. **Ablation.** Refit on the six numeric columns alone and compare cross-validated scores. If the score jumps, the 5,000 columns were noise carriers, not features. ## Why a wide one-hot block wrecks a distance One-hot encoding turns one variable into a block of columns, and a Euclidean distance treats every column as an equal, independent direction. So a single categorical with 5,000 levels arrives with 5,000 directions' worth of standing, while the six numeric columns that actually carry the signal arrive with six. Work through the raw case. With exactly one merchant per row, two rows either share the merchant — the block contributes 0 to squared distance — or differ, in which case they differ in exactly two columns and the block contributes 2. The distance decomposes as ``` D2 = (block: 0 or 2) + (numeric part) ``` That has two effects. First, if the numeric columns live on small scales, all different-merchant pairs sit at nearly the same distance and the near/far contrast collapses toward one: distance-weighted voting flattens, and any radius threshold becomes all-or-nothing. Second, any row sharing the merchant skips the offset entirely and leaps to the front of the neighbour list, however unlike the query it is numerically — the model degenerates into "look up other rows from the same merchant". Now the standardised case, which candidates often propose as the fix. Standardising a one-hot column divides by `sqrt(p(1-p))`, where `p` is that merchant's share of rows. For a merchant appearing in one row in five thousand, that divisor is about 0.014, so a mismatch on that column contributes roughly `1 / 0.0002`, in the thousands, to the squared distance. The block does not merely compete with the numeric features, it annihilates them, and the neighbour ordering becomes a near-arbitrary function of which rare merchants happen to be involved. Scaling is the right instinct for mixed units and the wrong instinct for a wide categorical block. ## The fixes, in order of directness - **Take the ID out of the metric.** Run the k-NN on the six numeric columns. This is the single change that most reliably restores contrast, and it costs only the information the ID carried — which the model was mishandling anyway. - **Give the block one variable's worth of weight.** If the merchant genuinely matters, weight the whole block so its total contribution is comparable to a single numeric feature, rather than letting 5,000 columns vote independently. - **Partition instead of encode.** Use the merchant as a grouping key — per-merchant or per-segment models, or a same-merchant filter applied before the numeric distance — so the ID selects the comparison set rather than distorting the geometry. - **Compress the ID into a small numeric summary.** Replacing thousands of indicator columns with one or two derived numbers keeps the information at a fraction of the dimensional cost. The choice of encoding scheme is a feature-engineering decision and belongs with that material, not with the metric. - **Change model family.** A gradient-boosted tree ensemble or a regularised linear model evaluates columns individually rather than summing them all into one distance, so a wide sparse block costs far less. If the merchant identity is genuinely predictive and high-cardinality, this is often the honest answer. ## What not to do Raising `k` does not help: it averages over a neighbour set that was chosen badly in the first place. Switching the distance function does not help either, because the problem is how many columns one variable occupies, not how the per-column differences are combined. Collecting more rows does not help, since the number needed to populate a 5,000-dimensional space is unreachable. ## Closing the loop Whatever you change, re-run the three diagnostics. The contrast ratio should rise, the share of neighbours sharing the query's merchant should drop back toward chance unless you deliberately partitioned on it, and cross-validated performance should improve. Reporting all three, rather than only the score, is what makes the fix defensible when someone asks why 5,000 features were thrown away.