Why is a similarity of 1.0 between two users with only 3 co-rated titles unreliable?

## The similarity is a statistic, not a measurement Every neighbourhood method ranks candidate neighbours by a number computed from the overlap between two rating profiles. It is easy to treat that number as if it were observed directly. It is not — it is an estimate computed from a sample whose size is the co-rating count `n`, and like any estimate it carries a variance that grows as `n` shrinks. With `n = 3`, a correlation of exactly 1.0 requires only that three deviations happen to line up in sign and rough proportion. Among millions of user pairs, an enormous number will manage it without sharing any real taste. With `n = 2`, correlation is essentially degenerate — two points define a line, so the magnitude is 1 whenever the two deviations are both non-zero. With `n = 1` it is undefined. Meanwhile a pair with 60 co-rated titles and a similarity of 0.7 has told you something you can act on. ## Why this is fatal rather than merely untidy Sparsity guarantees the problem dominates. In a rating community where the median user has rated a few dozen items out of hundreds of thousands, almost every user pair that overlaps at all overlaps on one, two or three items. So the distribution of raw similarities across all pairs is mostly made of low-`n` estimates, and the extreme values of that distribution — exactly the values a top-k selection picks — are almost entirely low-`n` noise. The consequence is a recommender whose neighbourhoods are random. Predictions become high-variance, unstable between rebuilds, and skewed toward whatever the accidental neighbours happened to like. It also fails silently: offline error metrics degrade a little, but the visible symptom is nonsense recommendations for exactly the users you most wanted to serve — the ones with short profiles. ## Shrinkage The principled correction is to pull every similarity toward zero by an amount that depends on how much evidence supports it: ``` sim_shrunk(u,v) = ( n_uv / (n_uv + lambda) ) * sim(u,v) ``` `n_uv` is the number of co-rated items and `lambda` is a positive constant. The multiplier is always between 0 and 1, so shrinkage never inflates anything. With `lambda = 25`: three co-ratings keeps 3/28, about 11% of the score; 25 co-ratings keeps half; 300 keeps 92%. The pair with 60 co-ratings at 0.7 now outranks the pair with 3 co-ratings at 1.0, which is the ordering you wanted all along. `lambda` is a hyperparameter, tuned like any other on a held-out split by the ranking or error metric you care about. Larger `lambda` demands more evidence before a neighbour is trusted. A related older device, significance weighting, multiplies by `min(n, N) / N` for some threshold `N` — a hard ramp instead of a smooth one, and it behaves similarly in practice. ## The other levers, and how they differ **A minimum overlap threshold.** Discard any pair with fewer than, say, 5 co-ratings outright. Blunt but effective, and worth having even alongside shrinkage as a floor. Its weakness is the cliff: a pair at 4 co-ratings contributes nothing while a pair at 5 contributes fully. **Neighbourhood size `k`.** This is a different knob and solves a different problem. Sweeping `k` from 5 to 200 on a sparse board-game rating community shows the classic curve: at very small `k` a single odd neighbour swings the prediction, so error is high and unstable; as `k` grows the average steadies and error falls; past some point you are averaging in weakly-similar neighbours and predictions collapse toward the item's mean, losing exactly the niche signal a recommender exists to find. Error typically bottoms out somewhere in the tens and rises slowly after. Shrinkage fixes *which* neighbours you trust; `k` controls *how many* you average — you need both, and shrinkage usually lets you use a smaller `k` safely. **Requiring the neighbour to have rated the target.** In user-based prediction, only neighbours who actually rated the target item contribute, so the effective neighbourhood is smaller than `k` for obscure items. Worth measuring: a nominal `k` of 50 can collapse to 3 real contributors on a tail item. ## Item-item is not exempt The same asymmetry appears between items. Two blockbuster titles may share 40,000 raters, while two niche titles share four. Without shrinkage, the tail items' similarity lists are noise, and since tail items are where recommendation value lives, that is the part you least want to leave broken. Applying the same `n / (n + lambda)` factor to item pairs is standard practice. ## The interview answer in one line "Because it is an estimate from three points. I would shrink it by co-rating count — `n/(n+lambda)` with `lambda` tuned on validation — so that a well-supported 0.7 outranks an accidental 1.0."