skip to content

Apache Lucene

3 roadmaps12 questions

The Java library that actually does the indexing and scoring underneath both Elasticsearch and Solr: segments, postings lists, analyzers, and the IndexWriter/IndexSearcher pair. Knowing this layer is how you explain why a document is not searchable until a refresh, or why segment merges cost so much I/O.

on this pageshow

explore

questions

12

How does Lucene delete a document from an immutable segment, and when is the disk space actually reclaimed?

level: middleimportance: must knowfreq 58%

answer

  1. No in-place edit means no true delete
  2. A bit per document, kept beside the segment
  3. Searches filter, they do not remove
  4. Statistics still count the dead
  5. Space comes back at merge time

basics

~20 s

Lucene never edits a segment, so a delete only marks the document in that segment's live-documents bitset and searches filter the hit out. The terms, stored fields and disk space go away only when a merge rewrites the segment without the dead documents.

solid answer

~50 s

`IndexWriter.deleteDocuments` resolves the term or query to internal document numbers and clears their bits in a per-segment **live docs** bitset, written as a small new generation file (`_1_1.liv`) alongside the untouched segment data. Searches consult that bitset and skip dead documents, so they vanish from results immediately, but their postings, stored fields and doc values are still on disk and still counted in the segment's term statistics — which is why document frequencies, and therefore scores, can drift slightly on a heavily-deleted index. `maxDoc` keeps counting them while `numDocs` does not. The space comes back only when a merge rewrites the surviving documents into a fresh segment and drops the old files. Lucene also supports **soft deletes**: instead of clearing a live-docs bit, the document is marked via a doc-values field and retained by a retention-aware merge policy, so it can still be read for operation history — this is what Elasticsearch uses to support replica recovery.

code

java · 3 lines
java
writer.deleteDocuments(new Term("id", "42"));
// document is filtered from results, but still on disk:
// reader.maxDoc() unchanged, reader.numDocs() decreased by 1

go deeper

for a junior

Know that a Lucene delete does not remove data: the document is marked and filtered out of results, and the space is only reclaimed later during a merge.

for a middle

Explain the live-documents bitset written as a new generation file, why updates are delete-plus-add, and the maxDoc versus numDocs distinction that exposes tombstones.

for a senior

Be ready to diagnose an index far larger than its live data — an update-heavy workload, starved merges, or retained soft deletes — and to choose between letting merging catch up and a targeted delete-expunging merge.

for a principal

Own the design question: an update-per-document-per-minute workload on an immutable-segment engine is a compaction problem, and the answer is usually to change the data model or the index lifecycle, not to tune deletes.

## Why a delete cannot be a delete A Lucene segment is written once and never modified, so there is no way to cut a document out of a postings list or a stored-fields block. The delete has to be recorded somewhere else and applied as a filter at read time. ## The live-documents bitset Each segment can carry a **live docs** bitset — one bit per internal document number in that segment, set when the document is alive. When you call `IndexWriter.deleteDocuments(Term)` or `deleteDocuments(Query)`, Lucene buffers the deletion, resolves it against each affected segment to a set of internal document numbers, and clears those bits. The result is written as a new small file with an incremented deletes generation, for example `_1_1.liv`, then `_1_2.liv`, and so on. The segment's real data files are not rewritten. Every query applies that bitset. `IndexSearcher` and the collectors it drives skip document numbers whose live bit is clear, so a deleted document disappears from results as soon as a reader is opened that sees the new deletes generation. ## What is still there The important consequence is that the document is only invisible, not gone: - Its postings entries remain in the term dictionary and postings lists, so scanning a term's postings still walks over them and pays the cost of skipping. - Its stored fields, doc values and term vectors are still on disk, occupying space. - It still contributes to the segment's collection and term statistics: document frequency for its terms, total document count, average field length. Scoring uses those statistics, so on an index where a large fraction of documents are deleted, BM25 scores are computed against a document population that includes the dead. - `IndexReader.maxDoc()` counts it; `IndexReader.numDocs()` does not. The gap between them is exactly the number of tombstoned documents, and it is the number to watch operationally. ## Updates are deletes `IndexWriter.updateDocument(term, doc)` is not an update. It deletes everything matching the term and adds a fresh document, which lands in whichever segment is currently being written. A workload that rewrites the same documents repeatedly therefore generates tombstones at exactly the rate it generates updates, and its index grows far beyond the live data size until merging catches up. ## When space actually returns Only a **merge** reclaims the space. A merge reads a set of segments, writes the still-live documents of all of them into one new segment with fresh internal numbering, and then deletes the input files once no reader is using them. Deleted documents are simply not copied. Lucene's `TieredMergePolicy` weighs deleted-document percentage when selecting merges and will schedule merges specifically to bring the index-wide deleted fraction back under its allowed threshold, so on a normally-configured index the space does come back on its own — eventually and asynchronously, not at delete time. If you need it back sooner on an index that has stopped changing, `IndexWriter.forceMergeDeletes()` rewrites segments whose deleted percentage is above a threshold. That is a targeted, cheaper alternative to a full `forceMerge(1)`, but it is still a rewrite of real data and should not be a routine background job. ## Soft deletes Lucene also supports **soft deletes**, configured with `IndexWriterConfig.setSoftDeletesField(...)` and applied with `IndexWriter.softUpdateDocument(...)`. Instead of clearing a live-docs bit, the delete is recorded by writing a marker doc-values field on the document. A retention-aware merge policy (`SoftDeletesRetentionMergePolicy`) then decides when such a document may finally be dropped by a merge, and readers can be wrapped so that soft-deleted documents are hidden from ordinary searches. The point of soft deletes is that the deleted documents remain physically readable for a while. That gives the layer above Lucene an ordered history of operations it can replay — which is how Elasticsearch supports bringing a lagging replica back into sync without copying whole segment files. The tradeoff is that retained soft-deleted documents keep occupying space and keep being carried through merges until the retention policy releases them. ## What this means in practice Deleting a large fraction of an index does not shrink it, and does not immediately speed anything up. Index size on disk must be read as live data plus tombstones plus merge headroom. If the deleted-document ratio is chronically high, the cause is usually an update-heavy workload or a merge policy that is not keeping up because merges are starved of I/O — not something you fix by deleting more carefully.

  • How do you tell how much of an index is tombstoned rather than live?
    Compare the reader's maxDoc with numDocs: maxDoc counts every document slot ever written in the currently-live segments, numDocs excludes the deleted ones, and the difference is the tombstone count. Per-segment tooling exposes the same figures alongside each segment's size, so you can see which segments are carrying the dead weight.
  • Why can scores shift slightly after a big merge even though no documents changed?
    Deleted documents keep contributing to term statistics — document frequency and average field length — until the segment holding them is merged. A merge drops them, so the collection statistics that BM25 uses change, and computed scores move a little. Statistics are also per-segment, so relevance can differ across segments before merging evens things out.
  • Is deleting by query more expensive than deleting by term?
    Yes. A delete by term can be resolved with a single term-dictionary lookup per segment, while a delete by query must execute that query against every segment to enumerate matching document numbers. Both then only clear bits, but the resolution step for a broad query can be as expensive as running the search itself.

saying these in an interview costs you the question

  • Says deleting documents frees disk space right away
  • Thinks the postings entries for a deleted document are removed
  • Ignores that deleted documents still affect term statistics
  • Confuses numDocs and maxDoc, or never mentions the gap
  • Treats an update as an in-place modification of the document

context

resolving links…

open as a page

Why are Lucene index segments immutable, and what does that design buy the search engine?

level: middleimportance: must knowfreq 68%

basics

~20 s

Lucene writes each segment once and never modifies it. Immutability lets readers share segments lock-free, enables aggressive write-once compression and per-segment caching, and makes indexing append-only. The cost is that deletes become tombstones and space returns only when segments merge.

open as a page

What are norms in a Lucene index, and how does BM25Similarity use them at query time?

level: middleimportance: must knowfreq 48%

basics

~20 s

Norms are a per-document, per-field length value written at index time, one byte per document under Lucene's default similarity. BM25Similarity reads that byte to normalize scores by field length, so a term match in a short field outranks the same match in a long one.

open as a page

In Lucene, what is the difference between postings, doc values, and stored fields for the same field?

level: middleimportance: must knowfreq 68%

basics

~20 s

Postings are the inverted index that maps a term to the documents containing it. Doc values are a columnar per-document copy used for sorting, faceting and scripts. Stored fields are the row-wise original bytes returned for the documents you display.

open as a page

How does a Lucene near-real-time reader make new documents searchable without calling IndexWriter.commit()?

level: middleimportance: should knowfreq 45%

basics

~20 s

Opening a reader with DirectoryReader.open(IndexWriter) flushes buffered documents into new segments and lets the reader see them without any fsync or new commit point. The documents become searchable in milliseconds but are not yet crash-durable.

open as a page

What does forceMerge(1) on a Lucene index actually cost, and when is it justified?

level: seniorimportance: should knowfreq 52%

basics

~20 s

forceMerge(1) rewrites every live document into a single segment: a full read and re-encode of the whole index, saturated CPU and I/O, up to roughly double the disk space while it runs, and a cold page cache afterwards. It is justified only on an index that has stopped receiving writes.

open as a page

How does Lucene's TieredMergePolicy decide which segments to merge?

level: seniorimportance: should knowfreq 42%

basics

~20 s

TieredMergePolicy sorts segments by size, works out how many segments of each size tier the index is allowed, and while the index is over budget picks the best-scoring candidate group of similarly-sized segments to merge — favouring low size skew and reclaiming deleted documents.

open as a page

Why does Lucene index numeric fields as BKD points rather than as terms in the inverted index?

level: seniorimportance: should knowfreq 38%

basics

~20 s

Numbers make terrible terms: encoding them for range search either explodes the term dictionary or forces long term enumerations. Lucene instead writes numerics into a balanced, block-oriented k-d tree on disk, where a range query skips whole leaf blocks that fall entirely inside or outside the bounds.

open as a page

How does Lucene's block-tree term dictionary use an FST, and what does that buy at query time?

level: seniorimportance: should knowfreq 40%

basics

~20 s

Lucene sorts terms and writes them in prefix-sharing blocks to the .tim file, then indexes those blocks with a finite state transducer in .tip that maps a term prefix to the block's file offset. The FST is tiny and memory-mapped, so term lookup and prefix or fuzzy enumeration touch very little data.

open as a page

How does Lucene compress stored fields, and what does choosing BEST_COMPRESSION trade away?

level: seniorimportance: should knowfreq 34%

basics

~20 s

Lucene groups consecutive documents into chunks and compresses each chunk as a unit, so fetching one document decompresses its whole chunk. BEST_SPEED uses fast LZ4; BEST_COMPRESSION uses a higher-ratio DEFLATE-family mode that shrinks the index but makes retrieval and merging measurably slower.

open as a page

How would you set a Lucene index's merge and commit strategy for continuous heavy ingest with low search latency?

level: principalimportance: should knowfreq 30%

basics

~20 s

Treat merge throughput as provisioned capacity, not a tuning afterthought. Size the RAM buffer so flushes produce reasonably large segments, reopen readers on a bounded interval rather than per write, commit on a cadence matched to an external durability log, and give merges enough I/O headroom to avoid indexing stalls.

open as a page

In a multi-terabyte Lucene index, how would you budget which per-field structures each field keeps?

level: principalimportance: should knowfreq 26%

basics

~20 s

Start from a written query contract: for each field, which of match, phrase, score, sort, aggregate, highlight and return it must support. Enable only the structures those verbs require, measure the per-structure file sizes on a representative sample, and treat every choice as reindex-only.

open as a page