skip to content

Why can BM25's IDF term go negative, and what do implementations do about it?

level: seniorimportance: nice to knowfreq 25%

answer

  1. a logarithm below one
  2. depends on how many documents hold the term
  3. the tipping point is half the collection
  4. matching a term would then hurt
  5. a plus one inside the log fixes it

basics

~20 s

The classic probabilistic IDF is a log-odds ratio that falls below zero once a term appears in more than half the collection, meaning a matching document is penalised for containing a very common word. Implementations add one inside the logarithm or floor the value.

solid answer

~50 s

BM25's IDF derives from the probabilistic relevance framework as `log((N - n + 0.5) / (n + 0.5))`, where N is the collection size and n the number of documents containing the term. When n exceeds roughly half of N, the numerator drops below the denominator, the ratio falls under 1, and the logarithm turns negative. The consequence is perverse: for a term like "the" in an ordinary corpus, a document that contains it scores *lower* than an otherwise identical document that does not, so matching more of the query hurts. Real implementations avoid this. The common fix is to add 1 inside the logarithm — `log(1 + (N - n + 0.5) / (n + 0.5))` — which is strictly positive for all n and decays smoothly toward zero for ubiquitous terms. Others clamp the value at zero or a small epsilon. Stopword removal reduces how often the case arises but does not eliminate it, since domain-specific words can be near-ubiquitous within one corpus.

code

python · 12 lines
python
import math

def idf_classic(n_docs, df):
    # goes negative once df > n_docs / 2
    return math.log((n_docs - df + 0.5) / (df + 0.5))

def idf_clamped(n_docs, df):
    # the +1 keeps the argument above 1, so the result is always positive
    return math.log(1 + (n_docs - df + 0.5) / (df + 0.5))

idf_classic(1000, 900)   # negative: containing the term lowers the score
idf_clamped(1000, 900)   # small and positive: the term is simply near-worthless

go deeper

for a junior

This is beyond the expected level. It is enough to know that IDF falls as a term becomes more common and that engines make sure very common terms contribute almost nothing.

for a middle

Be able to say that BM25's IDF is a log-odds expression rather than a plain ratio, and that its argument can fall below one, which is where the negative value comes from.

for a senior

Derive the half-collection threshold, describe the perverse ranking effect on a disjunctive query, and name the plus-one-inside-the-log mitigation and its alternatives.

for a principal

Recognise the general lesson: a component derived under one set of modelling assumptions can misbehave when embedded in a different objective, and scoring functions therefore need guard rails and corpus-aware review, not just correct derivations.

## Where the formula comes from BM25's IDF is not "log of N over document frequency" chosen by intuition. It is a log-odds expression from the probabilistic relevance framework, which asks how much more likely a term is to appear in a relevant document than in a non-relevant one. With no relevance feedback available, the estimate reduces to ``` idf(t) = log( (N - n + 0.5) / (n + 0.5) ) ``` where N is the number of documents in the collection and n is the number containing t. The 0.5 terms are smoothing, keeping the expression finite when n is 0 or N. ## Why it crosses zero A logarithm is negative when its argument is below 1. Solve for that: ``` (N - n + 0.5) / (n + 0.5) < 1 N - n + 0.5 < n + 0.5 N < 2n n > N / 2 ``` So the IDF of any term appearing in more than half of the documents is negative. This is not a bug in the derivation — under the log-odds reading, a term more common in the collection than not is genuinely weak evidence, and the model says so. The problem is what a negative weight does inside a *ranking* function. ## The perverse consequence Consider a corpus of English prose where "the" appears in 99% of documents, and a query `the quick fox`. With a raw negative IDF for "the", a document containing all three terms accumulates a negative contribution from "the", while a document containing only `quick fox` does not. The second document therefore outranks the first, purely for *lacking* a query term. Matching more of the query made the document worse. Worse, the effect scales with term frequency. Since the negative IDF multiplies a positive saturating frequency factor, a document that repeats the ubiquitous term is dragged down further than one mentioning it once. A perfectly ordinary document can be pushed below documents that match less of the query. This matters more than it first appears in multi-field or domain-specific corpora. In a corpus of legal contracts, "agreement" may appear in every document. In a support ticket index, "error" might. These are not stopwords in the general language, so no standard stopword list removes them, but their document frequency inside that particular collection exceeds the half mark. ## How implementations handle it **Add one inside the logarithm.** The widely used form is ``` idf(t) = log( 1 + (N - n + 0.5) / (n + 0.5) ) ``` Because the ratio is always positive, the argument is always above 1 and the result is always positive. As n approaches N the ratio approaches zero and the IDF decays smoothly toward log(1) = 0, so ubiquitous terms contribute nearly nothing rather than actively harming. This variant preserves the ordering of terms by rarity while removing the sign problem, and it is what several mainstream search libraries use. **Floor the value.** Some systems keep the classic expression but clamp the result at zero, or at a small positive epsilon. Clamping at exactly zero makes every term above the half-frequency threshold indistinguishable, which loses a little discriminating power among common terms; an epsilon preserves a token amount of ordering. **Remove the terms.** Stopword filtering during analysis prevents the most common function words from ever reaching the scorer. This helps but is not a solution: it depends on a static list, it is language-specific, and it cannot anticipate the domain terms that happen to be near-universal in one particular index. Many modern setups deliberately *keep* stopwords indexed — they are needed for phrase queries, and a well-behaved IDF already reduces their influence to almost nothing. ## Why not just accept negative scores One might argue that if the model says a term is negative evidence, the ranking should honour it. In practice this fails because retrieval is disjunctive: the candidate set is documents matching *at least one* query term. A negatively weighted term makes membership in the candidate set a liability, which is incoherent — the document that never matched at all is unaffected, while the one that matched is punished. The pathology is a consequence of applying a relevance-odds estimate designed for a fixed document set to an open ranking problem. ## What interviewers listen for That you can state the threshold (a term in more than half the collection) and derive it from the log-odds ratio, describe the perverse ranking effect concretely, and name at least one real mitigation — the plus-one inside the logarithm being the canonical answer. Knowing that stopword removal is a mitigation but not a fix separates a good answer from a memorised one.

  • At what document frequency does the classic probabilistic IDF cross zero?
    When the term appears in more than half the collection. Setting the log argument (N - n + 0.5)/(n + 0.5) below 1 gives N < 2n, so n greater than N/2 makes the IDF negative. The smoothing constants of 0.5 cancel exactly, which is why the threshold is the clean half-collection mark.
  • Does removing stopwords eliminate the negative IDF problem?
    No, it only reduces how often it arises. Stopword lists are static and language-specific, while the condition depends on this collection's statistics. A domain term like "agreement" in a contract corpus or "error" in a log index can exceed half of all documents while appearing on no stopword list. The scoring-side mitigation is the real fix.
  • What is the downside of clamping a negative IDF to exactly zero rather than using the plus-one form?
    Clamping flattens every term above the threshold into the same zero weight, so terms in 51% and 99% of documents become indistinguishable and contribute nothing at all. The plus-one form keeps them ordered by rarity with small positive weights, which preserves a little discriminating power among common terms at no extra cost.

saying these in an interview costs you the question

  • Says IDF is negative when a term is rare rather than common
  • Cannot state the more-than-half-the-collection threshold
  • Claims stopword removal fully solves the problem
  • Believes negative IDF is harmless because scores only need ordering
  • Thinks the fix is to take an absolute value of the IDF

context

resolving links…