Your HNSW build takes 11 hours on one machine — how do you plan rebuilds without downtime?

## Where 11 hours goes HNSW build cost is not mysterious. Every inserted vector runs the same beam search a query would run, with the beam set to efConstruction, to find its candidate neighbours, then prunes them to M links with a diversity heuristic. Cost per insert therefore scales with efConstruction and M, and total cost scales with the corpus size — with the per-insert cost itself creeping up as the graph grows. A generous efConstruction chosen to maximise recall is often the single largest contributor, and it is a choice someone made deliberately, once, for good reasons. The build is also memory-bound and largely single-machine: the graph is a shared mutable structure, so insertion parallelises across threads within a process reasonably well but does not distribute across machines for a single index. That is the constraint the plan has to route around. ## Lever one: rebuild less often The cheapest 11-hour build is the one you do not run. HNSW supports incremental insertion — new vectors join a live index without any training or rebuild step — so ordinary corpus growth is not a rebuild trigger. Be explicit about what genuinely is: - **A structural parameter change.** M and efConstruction are baked into the graph at insert, so changing either requires a full rebuild. This is why being generous with efConstruction on the first build is good economics. - **Re-embedding.** A new embedding model, or a dimensionality change, invalidates every stored vector. The re-embedding job usually dwarfs the index build itself. - **Tombstone accumulation.** Deletes are typically marked rather than removed, so past a measured ratio the index needs compaction to reclaim memory and traversal quality. - **Corruption or drift in the pipeline.** Occasionally you rebuild simply to re-establish a known-good state. Everything else — daily inserts, updates modelled as delete-plus-insert, filter metadata changes — should be handled online. ## Lever two: shard so builds run in parallel If one index cannot be built by many machines, build many indexes. Partition the corpus, give each shard its own index on its own machine, and have the query path fan out to all shards and merge the results by distance. Twenty shards over an 80M-row catalogue turn an 11-hour serial build into a sub-hour parallel one, and they also cap the blast radius of any single rebuild. The costs are real and worth naming. Fan-out means every query touches every shard, so tail latency becomes the max over shards rather than the mean, and one slow node hurts every request. Recall is preserved if each shard returns k and the merge takes the global best — the partitioning does not have to be semantically meaningful for correctness, and random partitioning is often the safest choice precisely because it keeps shard load even. Operationally you now have N indexes to build, monitor and roll out rather than one. If deletion correlates with time, partition by time as well: an expired partition is dropped whole rather than compacted, which removes an entire class of rebuild. ## Lever three: build offline, swap atomically Never mutate the serving index into the new shape. The pattern is blue-green: 1. Build the new index on separate capacity, from the source of truth, with the new parameters. 2. Warm it — traversal is random-access, and a cold index has a page-fault-dominated latency profile that looks like a regression. 3. Canary it against mirrored production traffic. Compare recall against the live index and against a brute-force ground truth on a query sample, and compare latency distributions, not just means. 4. Flip the pointer atomically behind the query path, so no request ever sees a half-built index. 5. Keep the old index resident and serviceable long enough to roll back with a second pointer flip. Steps 4 and 5 are why the memory plan must include a double-footprint window. Sizing a box to fit exactly one index means the first rebuild is an outage. Deltas need handling too: writes that arrive during the 11 hours must be captured and applied to the new index before the swap, usually by replaying a change log from the build's start watermark. Systems that already use segmented, incrementally-merged indexes get this for free, which is a strong argument for that architecture under continuous write load. ## The judgement call There is no single right answer here, and an interviewer is listening for the tradeoff rather than a recipe. Sharding buys build parallelism and blast-radius containment at the cost of fan-out tail latency and operational multiplicity. Blue-green buys safety at the cost of double capacity. Incremental inserts avoid rebuilds but let graph quality drift slowly as the data distribution moves away from what the graph was built for. A smaller efConstruction shortens the build but caps recall permanently for that index generation. Which you pick depends on write rate, freshness requirement, recall target and budget — and the strongest answers say so explicitly, then commit to a shape and name what they would measure to know it was the wrong call. ## What interviewers listen for The separation of concerns: what forces a rebuild versus what does not, how to parallelise a build that does not distribute, and how to deploy an index the way you would deploy a binary — built elsewhere, canaried, swapped atomically, rollback retained. Anyone who says "reindex overnight" without a swap strategy has not run one of these under load.