In a multi-terabyte Lucene index, how would you budget which per-field structures each field keeps?

## Framing the problem At multi-terabyte scale, index size is not a disk-cost question — disk is the cheapest part. It is a **cache-residency** question. Search latency is dominated by whether the structures a query touches are in the file-system cache. Every unnecessary structure competes for that cache with the structures that matter, so trimming the schema is a latency intervention as much as a storage one. It also compounds: bigger segments make merging slower, snapshots larger, recovery longer and replica rebuilds more expensive. ## The per-field cost model For every field, ask which of these capabilities are genuinely required, and note which structure each one implies: | Capability the product needs | Structure it requires | |---|---| | match a term or a filter | postings | | phrase / proximity / span matching | postings **with positions** | | offset-based highlighting | postings with offsets, or term vectors | | relevance scoring by length | norms | | sort, facet, group, script access | doc values | | numeric, date or geo range | points | | return the value to the caller | stored fields | | term-based similarity, some highlighters | term vectors | The two most common wins are at the top and the bottom of that table. **Positions** are frequently paid for on fields nobody phrase-searches; dropping to a frequencies-only or docs-only `IndexOptions` removes an entire postings file's worth of data on a large text field. **Term vectors** are frequently enabled once, for a highlighting experiment, and never removed. The cheap wins are smaller but free: norms on filter-only fields, doc values on fields never sorted or aggregated, stored copies of fields the API never returns. ## Deciding rather than guessing A principal-level answer has a method, not a list of tips. **1. Write the query contract.** For each field, enumerate the verbs the product actually performs against it. Not "someone might want to sort on this one day" — what is on the roadmap and what is in the API. This document is the artefact the schema is derived from, and the thing you point at when someone asks why a field is not sortable. **2. Derive the structures mechanically.** With the contract in hand the schema stops being a matter of taste; each verb turns on exactly one structure. **3. Measure on a sample.** Build a representative sample — real data, real analysis chain, enough documents that ratios stabilise — force it into a single segment, and list file sizes by extension. Postings, term dictionary, doc values, points, stored fields, norms and term vectors each live in their own extensions, so the breakdown is directly readable. Toggle one structure, rebuild, compare. Tools that inspect a Lucene index directly, or the engine's own disk-usage reporting, do this without hand-rolling anything. **4. Model the whole lifecycle, not the steady state.** A structure's cost is paid again on every merge, in every snapshot, and during every replica recovery. A 20% index reduction shortens all of those. **5. Decide the reversibility budget.** Which decisions do you want to be able to undo, and what would undoing them cost? ## The irreversibility constraint This is what separates a senior answer from a principal one. Every structure listed above is written at index time into an immutable segment. There is no operation that adds doc values to existing documents, turns norms back on, or starts writing positions for text already indexed. Merging only rewrites what is already there; it does not synthesise missing structures. So the real leadership decision is not the schema itself — it is **how expensive a wrong schema is to correct**. If reindexing the corpus is a routine, automated, well-rehearsed operation, you can trim aggressively and reverse a bad call in a day. If reindexing is a multi-week project nobody has run recently, every trim is a gamble and the rational move is to keep more than you need. Investing in the reindex pipeline is what buys the freedom to optimise the schema at all, and that inversion is the point worth making out loud. ## Tiering instead of a single answer One schema for all data is usually the wrong shape at this scale. Data ages, and its query contract ages with it: - **Hot data** — full capability, fast stored-fields compression, everything the product might need on the interactive path. - **Warm/cold data** — capabilities the product actually still exercises on old data, high-ratio stored-fields compression, positions and term vectors dropped if nothing phrase-searches or highlights history. Because the codec and the schema are fixed per segment, this is naturally implemented by writing new time-based indexes with different settings rather than by mutating anything. ## Failure modes to name - Enabling everything "to be safe", then discovering the index is several times the source data and no longer cache-resident. - Trimming a structure with no measurement, and discovering the saving was two percent while the lost capability was material. - Dropping positions on a field that a phrase query silently depends on, so a feature quietly stops matching rather than erroring. - Making the decision once, at project start, and never revisiting it as the query mix changes. ## The one-sentence version Derive the schema from an explicit query contract, prove each choice with measured per-extension sizes on a representative sample, tier the settings by data temperature, and — because every one of these choices is reindex-only — invest in making reindex cheap so the schema stays a decision you can revise rather than one you have to get right forever.