How does a Lucene near-real-time reader make new documents searchable without calling IndexWriter.commit()?

## Two different questions: visible, and durable Lucene deliberately separates *when new documents can be searched* from *when they are safe against a crash*. Confusing the two is the single most common mistake in this area. **Visibility** is controlled by which set of segments a reader is looking at. **Durability** is controlled by commits. ## Readers from a Directory see only commits `DirectoryReader.open(Directory)` finds the newest `segments_N` file in the index directory and opens exactly the segments listed there. A commit point is an immutable snapshot: the file names it references are guaranteed to exist and to be fsynced. A reader opened this way therefore sees a consistent, durable state, and nothing that has happened since the last commit. ## Near-real-time readers `DirectoryReader.open(IndexWriter writer)` returns a **near-real-time (NRT) reader**. Internally the writer flushes its in-memory document buffer to new segment files, resolves any buffered deletes so the live-docs bitsets are current, and gives the reader the writer's own current view of the segment list — including segments that are not named by any `segments_N` on disk. The result is a searchable view of everything indexed so far, typically available in milliseconds to tens of milliseconds rather than the hundreds of milliseconds an fsync-bearing commit costs. The catch is precisely the missing fsync: if the process dies, those uncommitted segments are not part of any commit point and are discarded on restart. Any system that promises write durability therefore layers its own write-ahead log underneath — this is exactly the role Elasticsearch's translog plays while its NRT reopen provides visibility. ## Reopening rather than opening Opening a reader from scratch re-reads every segment's metadata and rebuilds per-segment state. Because segments are immutable, that is nearly all wasted work: only the newly added segments and changed live-docs generations are actually new. `DirectoryReader.openIfChanged(oldReader, writer)` returns a new reader that shares the underlying per-segment readers with the old one and only opens what changed, or returns null if nothing changed at all. In practice you do not call this by hand on the search path. `SearcherManager` holds the current `IndexSearcher`, hands it out under reference counting through `acquire()`/`release()`, and swaps in a fresh one when `maybeRefresh()` is called. `ControlledRealTimeReopenThread` drives those refreshes on a schedule with two targets — a relaxed interval for ordinary refreshes and a tighter one when a caller is waiting for a specific write to become visible — which is how you build read-your-own-writes on top of near-real-time semantics without reopening on every request. ## What commit actually does `IndexWriter.commit()` flushes any buffered documents, fsyncs every file belonging to the segments that will be in the new commit, and then writes a new `segments_N` file naming them, where N is an incrementing generation. Writing that pointer file last is what makes the commit atomic: either the new `segments_N` exists and the commit took effect, or it does not and the previous commit is still the truth. `prepareCommit()` plus `commit()` splits this into the two phases explicitly, which lets a caller coordinate a Lucene commit with an external transaction. Old commit points are cleaned up by an `IndexDeletionPolicy`; the default keeps only the most recent one. Substituting `SnapshotDeletionPolicy` pins a commit so its files are not deleted, which is how a consistent backup of a live index is taken. Note that `IndexWriter.flush()` is not `commit()`: flushing pushes buffered documents into segment files but writes no commit point and provides no durability guarantee. ## The cost of reopening too often Near-real-time reopening is cheap but not free, and its real cost is downstream. Every reopen forces the writer to cut whatever is buffered into a segment, so a very aggressive reopen cadence produces a stream of tiny segments. Tiny segments mean more per-segment work on every query, more merge pressure, and colder caches, because each new segment arrives with nothing warmed. Reopening a hundred times a second on a heavy indexing stream is a reliable way to make search slower while chasing freshness nobody asked for. The usual answer is a reopen interval measured in hundreds of milliseconds to seconds, with a mechanism to wait for a specific write when a particular caller genuinely needs to see it.