How would you set a Lucene index's merge and commit strategy for continuous heavy ingest with low search latency?

## Frame the problem as three dials, not one The most common failure in this conversation is treating "make it fast" as a single knob. In a Lucene-based system there are three separable decisions, each with its own requirement and its own cost: - **Flush size** — how much is buffered in RAM before becoming a segment. Governs segment count at birth and therefore total merge work. - **Reopen interval** — how often a new reader is opened. Governs search freshness. - **Commit interval** — how often files are fsynced and a new `segments_N` is written. Governs crash durability. Stating them separately, and asking what the business actually requires for each, is most of the answer. ## Flush: make segments as large as you can afford Every segment born small must be merged upward, and each byte gets rewritten once per tier it climbs. The cheapest merge is the one never scheduled, so give `IndexWriter` as large a RAM buffer as heap safely permits. Be aware that anything forcing a flush — an NRT reopen, a commit, hitting a document-count limit — cuts the buffer short, so an aggressive reopen cadence silently defeats a generous buffer setting. If ingest is bursty, batching documents into larger `addDocuments` calls also helps by amortising per-call overhead. ## Reopen: bound the lag, do not chase zero Freshness has a price paid by everyone. Each reopen forces a flush, creating a segment; a hundred reopens a second creates a hundred segments a second, all of which must be searched until merged and all of which arrive with cold caches. Choose an interval that satisfies the actual product requirement — usually somewhere between a few hundred milliseconds and a few seconds — and implement it with `SearcherManager` plus a reopen thread, so all queries share one searcher generation. When a specific caller genuinely needs to see its own write, do not lower the global interval. Track the write's sequence number and let that caller wait for a searcher that includes it; `ControlledRealTimeReopenThread` supports exactly this with a relaxed interval for background refreshes and a tighter one for waiters. That way a rare requirement is paid for by the requests that have it. ## Commit: rarely, with a log underneath A commit fsyncs every file in the new commit point. On a large index that is expensive and it does nothing for visibility, which the NRT reader already handles. So commit on a slow cadence — measured in seconds to minutes depending on how much replay you can tolerate — and get per-operation durability from a write-ahead log maintained above Lucene, replaying anything after the last commit on restart. This is exactly the architecture the engines built on Lucene use, and it is the right answer to give: durability is not Lucene's commit's job alone. If you take backups, remember that pinning a commit with a snapshot deletion policy keeps its files alive; a long-held snapshot inflates disk usage because merged-away inputs cannot be deleted. ## Merges: provision, do not micro-tune The instinct is to tune `TieredMergePolicy`. Usually the defaults are fine and the real constraint is I/O. The signals to watch are segment count relative to the policy's budget, the deleted-document percentage, and merge-thread saturation. The failure mode to recognise is back-pressure: `ConcurrentMergeScheduler` stalls indexing threads once pending merges exceed its limit, so a storage device that cannot sustain merge writes presents as mysterious ingest latency, not as a merge alarm. When you do adjust: lowering `segmentsPerTier` buys query speed with more merge I/O and only makes sense if you have that I/O spare; raising `maxMergedSegmentBytes` yields fewer, larger segments at the cost of very long individual merges and more disk headroom; raising merge-thread count helps on SSD or NVMe and hurts on spinning disks where seeks dominate. Every one of these is a trade, and saying so explicitly is what distinguishes a considered answer. ## Separate the lifecycles The strongest structural move is to stop asking one index to be both write-optimised and read-optimised. Write into a current index; when it rolls over and stops taking writes, compact it once and leave it alone. That converts an unbounded steady-state merge cost into a one-off, schedulable cost, makes retention a file deletion instead of a mass delete-and-merge, and means the heavy compaction never competes with peak ingest. ## Capacity planning Budget three resources explicitly: I/O bandwidth for merges as a first-class line item alongside ingest and query; free disk equal to at least the largest permitted merge output, plus room for tombstones between merges; and heap for RAM buffers and per-segment structures. Set alerts on segment count, deleted percentage and merge backlog. Then verify the whole thing under sustained load rather than a burst — merge pressure is a steady-state phenomenon, and a benchmark that ends before the first large merge lands has measured nothing.