Why does BM25 usually rank better than raw TF-IDF on collections with long documents?

## The two failure modes of raw TF-IDF Raw TF-IDF weights a term as (some function of) its frequency in the document multiplied by its inverse document frequency. Two properties of that product cause trouble on real collections. **Unbounded frequency.** With undamped counts the weight is linear in occurrences: twenty mentions score ten times two mentions. Relevance does not work that way. The step from zero mentions to one is decisive, from one to two is meaningful, and from thirty to thirty-one is nothing at all. Linear weighting therefore rewards repetition far beyond its evidential value, which both flatters verbose documents and makes the model trivially gameable by keyword stuffing. Classic implementations damp with a logarithm or square root, which softens the problem without bounding it — the weight still grows forever, just more slowly. **Length handled as an afterthought.** Longer documents contain more of everything, so they accumulate higher term frequencies without being more relevant. The standard fix is cosine normalisation: divide by the document vector's Euclidean norm. That is a fixed, proportional correction applied identically to every collection. Empirical evaluation found it over-corrects — on real collections the probability that a long document is relevant is higher than the probability that cosine-normalised scoring will retrieve it, because long documents genuinely cover more ground and satisfy more needs. This observation produced pivoted length normalisation, which tilts the correction curve so long documents are not pushed down as hard. ## What BM25 changes BM25's per-term contribution is ``` idf(t) * (f * (k1 + 1)) / (f + k1 * (1 - b + b * dl / avgdl)) ``` **Saturation is built in.** As f grows the factor approaches k1 + 1 and stops. Each query term therefore has a bounded influence, no matter how often it is repeated. Consequences: repetition cannot buy a top rank; a document matching two distinct rare query terms beats one hammering a single term, because the bounded contributions of two terms sum higher than one term at its ceiling; and the parameter k1 lets you choose where the curve bends rather than accepting whatever shape a logarithm happens to give. **Length is inside the saturation denominator, and tunable.** The bracket `1 - b + b*dl/avgdl` equals 1 when b = 0 (length ignored) and dl/avgdl when b = 1 (fully proportional). It rescales the effective term frequency *before* saturation applies, rather than dividing the finished score. And because b is continuous, you can dial the correction to what a particular collection needs — near zero for short uniform fields like product names, at or above the usual default for long free text where verbosity is mostly padding. Cosine offers no such dial. **The IDF shape is derived, not chosen.** BM25's IDF comes from the probabilistic relevance framework — a log-odds expression in the number of documents containing the term — rather than from the intuition "rarer is better, take a log". In practice the curves are similar, but the derivation is what makes the parameters interpretable. ## The concrete scenario Take a query for a single term. Document A is a 300-token focused page mentioning it 4 times; document B is a 5000-token reference dump mentioning it 40 times. Under linear TF-IDF, B's frequency factor is ten times A's, and cosine normalisation may not be enough to overturn that if B's other weights are spread thin. Under BM25 with typical parameters, B's raw advantage is first cut by the length factor — its effective frequency is divided by roughly its length ratio to the average — and whatever survives lands on the flat part of the saturation curve where extra occurrences buy almost nothing. A wins, which matches what a user asking that query wants. ## Where BM25 does not help BM25 is still bag-of-words lexical matching. It does not know that "laptop" and "notebook computer" mean the same thing, it has no notion of word order beyond what a phrase query supplies, and it cannot tell a mention in a navigation menu from one in the main text. Vocabulary mismatch, semantics and document structure are outside the model entirely — they are addressed by analysis, field weighting, phrase and proximity handling, and separately by semantic retrieval. Nor does BM25 remove the need to measure: it is a better default, not a relevance strategy. ## Migration reality Swapping a system from TF-IDF to BM25 changes the absolute score scale as well as the ordering. Any threshold logic, score-based cutoffs or client code comparing scores across queries will misbehave; those were never safe, but a scoring change is when it shows. Score-based decisions should be replaced with rank-based ones before or during such a change. ## What interviewers listen for Naming the two mechanisms — bounded saturation and tunable length normalisation — and explaining the *consequence* rather than reciting the formula. A strong answer adds that BM25's advantages are largest exactly where document lengths vary widely, and smallest on collections of uniformly short fields where both models score much the same.