skip to content

Snapshots and Rebuilds

A vector index can be restored from a snapshot or rebuilt from sources kept elsewhere, and a rebuild that re-embeds can take hours. Interviewers ask which you would bet recovery on.

on this pageshow

questions

6

Why does 'we can always rebuild the vector index from source documents' mean a recovery time of hours, and what would you keep to skip re-embedding?

level: middleimportance: must knowfreq 42%

answer

  1. which stage dominates the rebuild
  2. tokens divided by throughput
  3. source quotas and index build too
  4. keep vectors with text and version

basics

~20 s

Rebuilding from source re-reads, re-parses and re-embeds every chunk before the index is built, and embedding millions of chunks under a throughput limit takes hours. Storing each chunk's vector, text and model version durably cuts a rebuild to loading plus index build.

solid answer

~50 s

A rebuild from source repeats the whole ingestion path: read every document from its sources, often under their request quotas, parse and chunk it, embed every chunk, then bulk-load and build the index. Embedding usually dominates: a scenario corpus of 20 million chunks at 500 tokens each is 10 billion tokens, which at a 10-million-tokens-per-minute limit is about 17 hours before the build starts. So I keep the expensive outputs: each chunk's vector, ID, text, metadata and exact model version, exported to cheap durable storage as they are written. A rebuild then loads those and pays only for the index build, which can still take hours for a large graph index, so I time it in a drill. The model version matters because vectors from two model versions must not be mixed in one index.

go deeper

for a junior

Recall the stages a rebuild from source repeats: read, parse and chunk, embed, load, build the index. Embedding every chunk again is the slow one.

for a middle

Do the arithmetic aloud: chunks times tokens divided by the throughput limit gives hours, then add the index build, and name what to store to skip the embed stage.

for a senior

Show you would time each stage in a drill, keep vectors with model version and text, and plan for a changed source, an expired credential or a drifted hosted model.

for a principal

Weigh the storage cost of keeping every vector against hours of downtime and repeat embedding spend, and decide whether the measured rebuild fits the recovery target.

## What "rebuild from source" actually repeats "We can always rebuild" sounds like a backup strategy, but a rebuild from source re-runs every stage of ingestion, in order, for the whole corpus: 1. **Read** every document from its source systems (a team wiki, a shared drive, a ticket tracker), usually through their APIs and under their request quotas. 2. **Parse and chunk** each document into retrieval units, which for scanned files can include slow text extraction. 3. **Embed** every chunk with the encoder, under its throughput limit or on your own accelerators. 4. **Load** the vectors and metadata into the store. 5. **Build** the approximate nearest-neighbour index over them. The **recovery time objective (RTO)** for the vector store is how long search may be down. If the rebuild is the plan, the rebuild's wall-clock time *is* the recovery time. ## Where the hours go With the stages pipelined, the slowest one sets the floor. Scenario figures for a mid-sized corpus: | Stage | What bounds it | Scenario time | |---|---|---| | Read sources | 2 million documents at a quota of 100 requests per second | 20,000 s, about 5.6 h | | Parse and chunk | CPU, more for scanned files | often minutes to a few hours | | Embed | 10 billion tokens at 10 million tokens per minute | 1,000 min, about 17 h | | Load | The store's write path | often under an hour | | Build the index | CPU for a large graph index | can be several hours | The arithmetic for the embed row: 20 million chunks × 500 tokens = 10 billion tokens; 10,000,000,000 ÷ 10,000,000 per minute = 1,000 minutes ≈ 16.7 hours. Real figures vary widely by provider, model and hardware, but the shape usually holds: re-embedding is the long pole, with source reads and the index build next. There is also spend: re-embedding the whole corpus is billed again. Sizing and running a planned re-embed is its own job; here it matters only as the cost of using sources as the recovery path. ## What to keep so a rebuild skips re-embedding The fix is to persist the expensive outputs, not just the inputs. For every chunk, keep: - the **vector** itself, at the precision the index was built from; - the **chunk ID** and the source document ID it came from; - the **chunk text** and the **metadata**, including the access list, so the rebuilt store can filter and answer; - the **encoder identifier and exact version**, plus preprocessing such as truncation length and normalisation; - the **chunker version**, so you know what produced the boundaries. Write these as columnar files to cheap durable object storage as chunks are written, or keep the pipeline's embedding cache keyed by content and model version. At 1,024 dimensions in 32-bit floats a vector is 4,096 bytes, so 20 million vectors take about 82 GB before text and metadata: real money, but small next to 17 hours of downtime. ## What a rebuild from stored vectors still costs Stored vectors remove the embed and read stages, not the whole bill: - **Index build** still runs, and for tens of millions of vectors on one machine it can take hours; time it in a drill rather than assume it is fast. - **Load** throughput depends on the store's bulk write path. - **Freshness:** the export reflects its last write, so deletes and permission changes since then must be replayed before serving. - **Encoder dependency:** queries still need the same model, so the stored vectors are only useful while that encoder is callable. ## Traps - Assuming the sources will still be there, unchanged: documents get deleted, retention policies purge history, and a connector's credentials expire. - Rebuilding through a hosted model whose name now points at a newer version, silently producing vectors in a different space from anything stored. - Sharing the source's request quota with its human users during a full re-read. - Storing vectors without the model version, which makes an export impossible to trust for a rebuild. - Never timing the rebuild, so the "plan" is an untested estimate.

  • Why not re-embed with a newer, cheaper model while you are rebuilding anyway?
    Because the recovery then becomes a model migration under pressure: thresholds, rerankers and evaluation baselines were tuned on the old model, and there is no time to validate the new one. Recover on the same model version first so behaviour matches what users had, then plan the migration as its own job with its own checks.
  • Stored vectors existed, yet the rebuild still took six hours. Where did the time go?
    Most likely the index build. Building a graph index over tens of millions of vectors is CPU-bound and can take hours on one machine, and bulk loading may be throttled by the store's write path. Time each stage in a drill; speeding up the build itself, for instance by building pieces in parallel, belongs to index-building work rather than to the recovery plan.

  • A rebuild is mostly the time to copy documents back.
  • Re-embedding is quick because each embedding call is fast.
  • Storing vectors without their model version is fine for a rebuild.
  • A rebuild from stored vectors is instant; nothing is left to compute.
  • The sources will still be there, unchanged, whenever a rebuild is needed.
open as a page

After restoring a vector store from a week-old snapshot, which deletions and permission revocations come back, and how do you re-apply them before serving?

level: middleimportance: must knowfreq 38%

basics

~20 s

Every chunk deleted and every permission revoked after the snapshot comes back, along with stale versions of updated documents. Keep an ordered journal of changes outside the store, replay it from the snapshot time with traffic closed, verify, then serve.

open as a page

A vector-store snapshot is intact, but the hosted embedding model that produced its vectors has been retired; what is that snapshot still worth?

level: seniorimportance: should knowfreq 25%

basics

~20 s

New text queries can no longer be embedded into the snapshot's vector space, so query search over it is lost. Its chunk text and metadata still seed a re-embed with a new model, and item-to-item similarity among stored vectors still works.

open as a page

An overnight ingest job wrote corrupt vectors into 2% of a vector index; how do you undo it without losing the day's good writes?

level: seniorimportance: should knowfreq 34%

basics

~20 s

Restoring last night's snapshot would discard the day's good writes and revive deletes, so repair surgically: find the bad job's records by their load ID, hide them, then rewrite only those IDs with correct vectors from kept copies or a fresh embed.

open as a page

As the lead, would you make snapshot restore or rebuild from kept sources your team's standing recovery bet for a vector store, and what decides it?

level: principalimportance: should knowfreq 30%

basics

~20 s

Bet on the path your drills prove meets the recovery time: snapshots usually restore fastest but revive deletes and stay tied to their encoder; rebuilds from sources are current but slow. Many teams keep both, plus stored vectors between.

open as a page

A sharded vector store snapshots each node on its own and keeps chunk text in a separate store; why can the restored system be inconsistent, and how do you prevent it?

level: seniorimportance: nice to knowfreq 22%

basics

~20 s

Copies snapshotted minutes apart freeze each shard and the text store at different moments, leaving half-present documents, mixed versions and vectors with no text. Take one cut with writes paused, or record each copy's log position and replay all to one point.

open as a page