What are norms in a Lucene index, and how does BM25Similarity use them at query time?

## What a norm actually is A norm in Lucene is a single value stored per document, per field, describing how "long" that field was when it was indexed. It exists so that scoring can discount matches in long fields. Without it, a query term appearing once in a book-length body would score the same as the same term appearing once in a short title, and relevance would collapse toward whichever documents happen to contain the most text. Norms are produced during inversion, not during search. As `IndexWriter` analyzes a field it accumulates a `FieldInvertState` — token count, position increments, overlap count, unique term count — and then calls `Similarity.computeNorm(FieldInvertState)`, which returns a `long`. That value goes through the `NormsFormat` into the segment's `.nvd` (data) and `.nvm` (metadata) files, alongside doc values in spirit: they are read per matching document as a forward iterator. ## The one-byte encoding `BM25Similarity.computeNorm` takes the field length — by default `state.getLength() - state.getNumOverlap()`, i.e. tokens minus those injected at the same position, which keeps synonym filters from inflating the apparent length — and compresses it with `SmallFloat` into a **single byte**. That is a deliberately lossy, roughly logarithmic bucketing: short lengths are represented precisely, long ones coarsely. A 900-token field and a 1000-token field very often land in the same bucket and receive an identical length penalty. Candidates are frequently surprised by this. They expect a precise integer length and try to explain a scoring difference by a few tokens. The right answer is that below a certain granularity the norm cannot distinguish them at all. The payoff of the lossy encoding is size: one byte per document per field, decoded through a small lookup table, cheap enough to read for every scored hit. ## How BM25 consumes it Lucene's `BM25Similarity` scores a term as roughly: ``` idf * tf / (tf + k1 * (1 - b + b * dl / avgdl)) ``` with `idf = ln(1 + (docCount - docFreq + 0.5) / (docFreq + 0.5))`. Here `tf` is the term frequency from the postings, `dl` is the decoded norm, and `avgdl` is the average field length derived from segment-level statistics. The defaults are `k1 = 1.2` and `b = 0.75`. - `k1` controls **term-frequency saturation**: how quickly repeating a term stops helping. - `b` controls **how strongly length normalization applies**. At `b = 0` the norm is ignored entirely; at `b = 1` the penalty is fully proportional to length over average length. Both are read at query time from the `Similarity` instance, so they can be changed without touching the index. The norm itself cannot. ## Index-time versus query-time, and why it matters This split is the interview point. If you replace the similarity with one whose `computeNorm` encodes something different — a different length definition, a different discretization — documents already in the index still carry norms written under the old rule, and their scores will be wrong until those segments are rewritten. Changing `k1`/`b` on `BM25Similarity`, by contrast, takes effect on the next query for every document. ## Omitting norms A field can be indexed with norms disabled (`FieldType.setOmitNorms(true)`, exposed as a schema flag by the engines built on Lucene). Three things follow: 1. Roughly one byte per document per field is saved. On a billion-document index across a dozen analyzed fields that is a meaningful, though not dramatic, saving. 2. Length normalization disappears for that field: BM25 behaves as if every document's field were of average length. 3. Fields that were never scored — a status code, a tenant ID, anything used purely as a filter — lose nothing, because filter clauses do not score. Because norms are written at index time, flipping the flag only affects newly written segments; existing documents keep whatever they were indexed with, and mixed segments will score inconsistently until a reindex. ## Common failure modes - A field that is only filtered on still carries norms because nobody turned them off; pure waste, but harmless to relevance. - A synonym filter is applied and someone expects the field to look longer; by default overlapping tokens are discounted, so it does not. - Someone concludes from a scoring difference of a few points that norms track exact token counts. They do not — the encoding is bucketed. - Someone disables norms on the main body field to save space and is then surprised that long documents dominate results.