Why does 'we can always rebuild the vector index from source documents' mean a recovery time of hours, and what would you keep to skip re-embedding?
answer
- which stage dominates the rebuild
- tokens divided by throughput
- source quotas and index build too
- keep vectors with text and version
basics
~20 sRebuilding from source re-reads, re-parses and re-embeds every chunk before the index is built, and embedding millions of chunks under a throughput limit takes hours. Storing each chunk's vector, text and model version durably cuts a rebuild to loading plus index build.
solid answer
~50 sA rebuild from source repeats the whole ingestion path: read every document from its sources, often under their request quotas, parse and chunk it, embed every chunk, then bulk-load and build the index. Embedding usually dominates: a scenario corpus of 20 million chunks at 500 tokens each is 10 billion tokens, which at a 10-million-tokens-per-minute limit is about 17 hours before the build starts. So I keep the expensive outputs: each chunk's vector, ID, text, metadata and exact model version, exported to cheap durable storage as they are written. A rebuild then loads those and pays only for the index build, which can still take hours for a large graph index, so I time it in a drill. The model version matters because vectors from two model versions must not be mixed in one index.
go deeper
Recall the stages a rebuild from source repeats: read, parse and chunk, embed, load, build the index. Embedding every chunk again is the slow one.
Do the arithmetic aloud: chunks times tokens divided by the throughput limit gives hours, then add the index build, and name what to store to skip the embed stage.
Show you would time each stage in a drill, keep vectors with model version and text, and plan for a changed source, an expired credential or a drifted hosted model.
Weigh the storage cost of keeping every vector against hours of downtime and repeat embedding spend, and decide whether the measured rebuild fits the recovery target.
## What "rebuild from source" actually repeats "We can always rebuild" sounds like a backup strategy, but a rebuild from source re-runs every stage of ingestion, in order, for the whole corpus: 1. **Read** every document from its source systems (a team wiki, a shared drive, a ticket tracker), usually through their APIs and under their request quotas. 2. **Parse and chunk** each document into retrieval units, which for scanned files can include slow text extraction. 3. **Embed** every chunk with the encoder, under its throughput limit or on your own accelerators. 4. **Load** the vectors and metadata into the store. 5. **Build** the approximate nearest-neighbour index over them. The **recovery time objective (RTO)** for the vector store is how long search may be down. If the rebuild is the plan, the rebuild's wall-clock time *is* the recovery time. ## Where the hours go With the stages pipelined, the slowest one sets the floor. Scenario figures for a mid-sized corpus: | Stage | What bounds it | Scenario time | |---|---|---| | Read sources | 2 million documents at a quota of 100 requests per second | 20,000 s, about 5.6 h | | Parse and chunk | CPU, more for scanned files | often minutes to a few hours | | Embed | 10 billion tokens at 10 million tokens per minute | 1,000 min, about 17 h | | Load | The store's write path | often under an hour | | Build the index | CPU for a large graph index | can be several hours | The arithmetic for the embed row: 20 million chunks × 500 tokens = 10 billion tokens; 10,000,000,000 ÷ 10,000,000 per minute = 1,000 minutes ≈ 16.7 hours. Real figures vary widely by provider, model and hardware, but the shape usually holds: re-embedding is the long pole, with source reads and the index build next. There is also spend: re-embedding the whole corpus is billed again. Sizing and running a planned re-embed is its own job; here it matters only as the cost of using sources as the recovery path. ## What to keep so a rebuild skips re-embedding The fix is to persist the expensive outputs, not just the inputs. For every chunk, keep: - the **vector** itself, at the precision the index was built from; - the **chunk ID** and the source document ID it came from; - the **chunk text** and the **metadata**, including the access list, so the rebuilt store can filter and answer; - the **encoder identifier and exact version**, plus preprocessing such as truncation length and normalisation; - the **chunker version**, so you know what produced the boundaries. Write these as columnar files to cheap durable object storage as chunks are written, or keep the pipeline's embedding cache keyed by content and model version. At 1,024 dimensions in 32-bit floats a vector is 4,096 bytes, so 20 million vectors take about 82 GB before text and metadata: real money, but small next to 17 hours of downtime. ## What a rebuild from stored vectors still costs Stored vectors remove the embed and read stages, not the whole bill: - **Index build** still runs, and for tens of millions of vectors on one machine it can take hours; time it in a drill rather than assume it is fast. - **Load** throughput depends on the store's bulk write path. - **Freshness:** the export reflects its last write, so deletes and permission changes since then must be replayed before serving. - **Encoder dependency:** queries still need the same model, so the stored vectors are only useful while that encoder is callable. ## Traps - Assuming the sources will still be there, unchanged: documents get deleted, retention policies purge history, and a connector's credentials expire. - Rebuilding through a hosted model whose name now points at a newer version, silently producing vectors in a different space from anything stored. - Sharing the source's request quota with its human users during a full re-read. - Storing vectors without the model version, which makes an export impossible to trust for a rebuild. - Never timing the rebuild, so the "plan" is an untested estimate.
- Why not re-embed with a newer, cheaper model while you are rebuilding anyway?Because the recovery then becomes a model migration under pressure: thresholds, rerankers and evaluation baselines were tuned on the old model, and there is no time to validate the new one. Recover on the same model version first so behaviour matches what users had, then plan the migration as its own job with its own checks.
- Stored vectors existed, yet the rebuild still took six hours. Where did the time go?Most likely the index build. Building a graph index over tens of millions of vectors is CPU-bound and can take hours on one machine, and bulk loading may be throttled by the store's write path. Time each stage in a drill; speeding up the build itself, for instance by building pieces in parallel, belongs to index-building work rather than to the recovery plan.
- A rebuild is mostly the time to copy documents back.
- Re-embedding is quick because each embedding call is fast.
- Storing vectors without their model version is fine for a rebuild.
- A rebuild from stored vectors is instant; nothing is left to compute.
- The sources will still be there, unchanged, whenever a rebuild is needed.