In an Elasticsearch mapping, how do norms and index_options change BM25 scoring on a field?

## Norms: the length half of BM25 BM25's denominator contains `(1 - b + b * dl / avgdl)`, where `dl` is the length of this field in this document. That length is not recomputed at query time — it is written at index time into a per-document structure called **norms** (a single lossily-encoded byte per document per field). Norms are why a three-word title matching "kotlin" outranks a 5,000-word body matching it once. Setting `"norms": false` in the field mapping stops writing them. The consequences: - **Length normalisation disappears.** Every matching field is treated as if it were of average length, so the `b` parameter no longer has any effect on that field. Scoring reduces to term frequency and idf. - **You save space and heap.** One byte per document per field sounds trivial, but on an index with hundreds of fields and billions of documents it is not, and norms are read during scoring. - **It is a one-way door.** Norms can be disabled on an existing field with the update mapping API; existing norms are dropped lazily as segments are merged. Turning them back on only affects newly indexed documents, so restoring length normalisation for the whole corpus means reindexing. The standard candidates for disabling norms are fields you filter or aggregate on but never rank by — status flags, identifiers, log metadata. Note that `keyword` fields have norms off by default already, which is one reason a `term` query on a keyword field is driven almost entirely by idf. ## index_options: the frequency and position ladder `index_options` controls what the postings list stores for each term, and each rung is a superset of the one below: - **`docs`** — only which documents contain the term. Term frequency is unavailable, so BM25 treats every occurrence count as one: a document mentioning the term twenty times scores exactly like one mentioning it once. This is the default for `keyword` fields. - **`freqs`** — adds the number of occurrences per document, so BM25's tf component becomes meaningful and `k1` starts to matter. - **`positions`** — adds token positions. This is the default for `text` fields, and it is what phrase queries (`match_phrase`, `span` queries, phrase-aware `multi_match` modes) require. Without positions those queries cannot run on the field. - **`offsets`** — adds character offsets, used by the unified highlighter to highlight without re-analysing the field. Each rung costs disk and indexing time. Dropping a rarely-searched `text` field from `positions` to `freqs` is a real saving on a large index, but only if nothing ever issues a phrase query against it — and "nothing ever" tends to be a promise the product team has not made. ## How the two interact with BM25 Put side by side, the two parameters remove different halves of BM25: | Setting | What BM25 loses | |---|---| | `norms: false` | field-length normalisation; `b` has no effect | | `index_options: docs` | term frequency; `k1` has no effect, repeats do not help | | both | scoring collapses to idf alone | That last row is worth remembering: a `text` field with norms off and `index_options: docs` scores essentially like a boolean-with-idf match. If that is what you wanted, saying so via the `boolean` similarity is clearer, cheaper and reversible. ## Diagnosing it after the fact When relevance looks strangely flat — long and short documents interleaved, repeated terms not helping — check the mapping before touching the query. The explain output helps too: if the tf node reports the same value for documents with obviously different field lengths, norms are not there. Because the effects of these parameters are baked in at index time, the fix is often a reindex, which is why they deserve thought when the mapping is designed rather than after a relevance complaint. ## The sensible default Leave both alone unless you have a measured storage problem. The defaults — norms on and `positions` for `text`, norms off and `docs` for `keyword` — are already the right shape for the intended use of each type. Trim them on the fields you can name and justify: high-volume log fields, denormalised copies used only for filtering, and fields that exist purely as aggregation keys.