Ranking Models: TF-IDF & BM25

## The formula BM25 scores a document by summing a contribution per query term: ``` score(d, q) = SUM over t in q of idf(t) * (f * (k1 + 1)) / (f + k1 * (1 - b + b * dl / avgdl)) ``` where f is the number of occurrences of term t in document d, dl is the length of d in tokens, and avgdl is the mean document length across the collection. Two free parameters shape the curve: k1 and b. Everything else is corpus statistics. ## k1: how fast term frequency saturates Ignore length for a moment by setting b = 0. The term-frequency factor is then f(k1+1)/(f+k1). This is a hyperbola: it rises steeply from zero, then bends over toward a horizontal asymptote at k1 + 1. That asymptote is the crucial property — no matter how many times a term is repeated, its contribution is bounded. k1 chooses where the bend happens. Small k1 means the curve flattens almost at once: the first occurrence carries nearly all the weight, and the twentieth is barely different from the second. Large k1 straightens the curve, so the score keeps climbing with frequency and BM25 starts behaving more like raw TF-IDF. In the limit k1 -> infinity you recover linear term frequency; at k1 = 0 the factor collapses to exactly 1 for any f > 0, so the model becomes binary presence-weighted-by-IDF, which is essentially a probabilistic Boolean retrieval. The practical value of saturation is that it defends against repetition. A page that repeats a keyword forty times gains very little over a page mentioning it five times, so keyword stuffing does not buy a top rank, and a genuinely on-topic short document is not automatically beaten by a verbose one. ## b: how strongly length is normalised The denominator contains `k1 * (1 - b + b * dl/avgdl)`. Read the bracket alone: - at **b = 0** it is `1`, independent of dl — the length signal is switched off entirely; - at **b = 1** it is `dl/avgdl` — the correction is fully proportional, so a document twice the average length effectively has its term frequencies halved; - between, it interpolates linearly between those two regimes. A document longer than average therefore inflates the denominator and lowers the score, and a shorter-than-average one gets a boost. The intuition is that finding a term three times in a tweet is far stronger evidence than finding it three times in a book chapter. Note that b sits *inside* the saturation denominator rather than dividing the final score. That coupling is deliberate: length rescales the effective term frequency before saturation applies, rather than being a separate multiplier bolted on afterwards as it is in cosine-normalised TF-IDF. ## Defaults and when they are wrong k1 = 1.2 and b = 0.75 are the widely used defaults and are a reasonable starting point across a broad range of collections. They are not universal truths, and two situations regularly justify deviating. The first is a collection of **near-uniform short fields** — product names, person names, tags, identifiers. Length normalisation has almost nothing to correct there, and a strong b can produce odd effects where a one-token variant beats a natural title. Lowering b, sometimes to zero, is a defensible move. The second is **long, heterogeneous text** where verbosity genuinely correlates with padding: a high b pushes back on it. Conversely, in a corpus where longer documents genuinely are more useful — comprehensive reference pages against thin stubs — a lower b avoids punishing them for being thorough. k1 is adjusted less often. Raising it suits collections where repetition really is evidence of topicality; lowering it suits collections where any mention is as good as many, such as short metadata fields. ## Fields differ, so parameters may differ A single document often has parts with wildly different length distributions — a title of five tokens and a body of two thousand. Applying one b to both is a compromise. Engines commonly allow per-field similarity configuration so a title field can use a small b and the body a larger one; the details of that configuration are engine-specific, but the reasoning is the model's. ## Tuning discipline k1 and b should be changed only against a judgment set and a measurement, never on the strength of one query looking better. The gains from re-fitting them are usually modest compared with fixing analysis, field weighting or query construction, and it is easy to overfit two parameters to a handful of anecdotes. Grid search over a small range with held-out queries is the standard approach when it is worth doing at all. ## What interviewers listen for That you attach k1 to saturation and b to length — getting these backwards is the most common error — and that you can state the boundary behaviours: k1 = 0 gives binary scoring, b = 0 disables length normalisation, b = 1 makes it fully proportional. Knowing the defaults is a bonus, not the point.