How do you add popularity signals to search ranking without a rich-get-richer feedback loop?

## Three problems wearing one coat "Add popularity to ranking" hides three distinct issues: the **distribution** of the raw signal, its **share** of the final score, and the **causal loop** between exposure and the signal itself. Fixing only the first two produces a system that is well-behaved and permanently frozen. ## Problem one: the distribution Engagement counts are heavy-tailed. In most catalogues the top item has orders of magnitude more clicks than the median, so any linear use of the raw count means popularity, not relevance, decides the ranking. The fix is a saturating transform that compresses the tail: ``` pop = clicks / (clicks + k) # approaches 1; k sets where saturation begins pop = log(1 + clicks) / log(1 + clicks_p99) # log-compressed, normalised ``` The parameter `k` is a real design decision: it is roughly the count at which a document is considered half as popular as anything can be. Pick it from the distribution — near the median or the 75th percentile of engaged documents — not from taste. ## Problem two: the share Even a saturated signal must be bounded in how much it can move the ranking. Applying it as a bounded multiplier makes the cap explicit and auditable: ``` score = relevance * (1 + w * pop) # popularity can add at most w (e.g. 30%) ``` With `w = 0.3` you can state precisely what popularity is allowed to do, and a reviewer can check that claim. Compare that with an unbounded additive bonus, where nobody can say what the ceiling is. The same discipline applies to business signals — margin, promotions, stock-clearing — which should be their own capped multiplier, separately owned and separately reviewable, never blended silently into the relevance weights. ## Problem three: the feedback loop This is the one interviewers are really asking about. Clicks are not an unbiased measurement of quality, because **rank causes clicks**. Under the examination hypothesis, a click happens when a user both examines a position and finds the result relevant: ``` P(click | doc at rank r) = P(examine | r) * P(relevant | doc) ``` Examination probability falls steeply with rank. So a mediocre document at position 1 collects more clicks than an excellent one at position 8, gets a higher popularity score, ranks first again tomorrow, and the loop closes. Left alone, the ranking calcifies around whatever happened to be popular when the signal was switched on, and genuinely better documents can never accumulate the evidence needed to displace it. The mitigations, roughly in order of effort: **Normalise by position.** Estimate the examination probability per rank from your own logs, then divide observed clicks by the summed examination probability of the positions where the document was actually shown. That is inverse-propensity weighting, and it converts "clicks" into something much closer to "click-through rate given that it was seen". **Age the signal.** Use a decaying window — clicks in the last 30 days, or an exponentially-weighted count — so a document that was popular last year has to keep earning it. Without ageing, the loop is not just self-reinforcing, it is permanent. **Explore.** Reserve a small amount of traffic or a small perturbation of the ranking so documents outside the incumbent top-k get impressions. This can be an epsilon-greedy shuffle of positions near the boundary, a dedicated slot for under-exposed items, or a time-boxed exposure allowance for newly indexed documents. Exploration costs measurable short-term engagement and buys the ability to ever improve — treat that cost as a budget line, not an accident. **Handle cold start honestly.** A new document with zero clicks should not receive popularity zero; it has *no evidence*, which is different from *evidence of being bad*. Smooth toward a prior — the mean popularity of its category, for instance — so a new item starts at neutral and moves as evidence arrives. A Bayesian shrinkage of the form `(clicks + a*prior) / (impressions + a)` does this naturally and also stops a document with two clicks from two impressions looking like a 100% winner. **Normalise per segment.** A single global popularity prior drowns niche intents: the most-clicked item overall is not the right answer for a specialist query. Where you can, compute popularity conditioned on the query, the query cluster, or the category, so "popular *for this kind of query*" is what enters the score. ## What to monitor The loop is invisible in aggregate metrics — engagement often looks fine while diversity collapses. Watch the ones that expose calcification directly: the share of impressions going to the top 1% of documents, the median age of documents appearing in the top ten, the rate at which new documents ever reach page one, and the churn in the top-k for a fixed query set over time. If nothing ever changes position, the ranking has stopped learning.