An overnight ingest job wrote corrupt vectors into 2% of a vector index; how do you undo it without losing the day's good writes?
answer
- a full restore loses later writes
- which records did that job touch
- load ID and model version per record
- hide first, repair second
- rewrite only the damaged IDs
basics
~20 sRestoring last night's snapshot would discard the day's good writes and revive deletes, so repair surgically: find the bad job's records by their load ID, hide them, then rewrite only those IDs with correct vectors from kept copies or a fresh embed.
solid answer
~50 sA whole-store restore is the wrong tool: to fix 2% of records it rolls back every good write and every delete since the snapshot. I first stop the job and hide its output, for example with a query-time filter on the load ID every record should carry. Then I build the repair set: records the job created or overwrote, minus any that a later good write has already replaced. For each, I re-embed the stored chunk text with the correct encoder, or, where the text is unchanged and re-embedding is slow, take the previous vector from last night's snapshot loaded into a scratch instance or from the stored-vector export. Then I re-upsert, remove the filter and verify. If records carry no load ID, I have to find the damage from the vectors themselves, which is why I stamp lineage on every write.
code
json · 10 lines{
"id": "doc-4812#chunk-7",
"vector": [0.012, -0.087, 0.044],
"metadata": {
"source_doc_id": "doc-4812",
"load_id": "ingest-2026-10-16-nightly",
"embedding_model": "encoder-a@2026-03",
"written_at": "2026-10-16T23:41:07Z"
}
}go deeper
Know that one bad load does not justify throwing away a day of good writes, and that each record should say which job wrote it.
Explain why corrupt vectors fail silently: search still returns ranked results, only wrong ones, so detection needs lineage fields or checks on the vectors themselves.
Walk through contain, scope, repair and verify, including records a later good write replaced, a scratch instance for old vectors, and a fixed query set to confirm the fix.
Decide where surgical repair stops and a full restore starts, and make lineage fields, validation gates and staged loads a standard every pipeline must meet.
## Why a full restore is the wrong tool A **bad load** is an ingest run that wrote wrong data: here, vectors produced by a misconfigured encoder (the wrong model version, a dropped normalisation step, or empty text embedded in place of the chunk). Only 2% of records are affected, but the instinct is to restore last night's snapshot. That trades a small problem for three larger ones: - Every good write made since the snapshot is lost and has to be re-ingested. - Every delete and permission revocation since the snapshot comes back and has to be replayed. - Search is down, or degraded, for the whole restore. A whole-store restore is still the right call in some cases: - the damage is wide, or its extent cannot be determined; - records carry no lineage and the vectors give no reliable signal of which ones are bad; - the bad job also deleted records at scale, and no log of those deletes exists. For a bounded, identifiable 2%, surgical repair is faster and safer, because it touches only the records that are wrong and leaves every later good write and delete in place. ## Step 1: contain 1. **Stop the job** and any retries, and fix the cause before anything is re-run; re-running the same job unchanged writes the same corruption. 2. **Hide the damage.** If each record carries the load ID that wrote it, add a query-time filter excluding that ID. Results lose those chunks for a while, which is better than serving wrong neighbours. A filter on a small fraction of records usually costs little recall, though heavy filtering can starve an approximate search of candidates. ## Step 2: find the blast radius Not every record the job touched needs the same treatment. | Record class | How to find it | Repair | |---|---|---| | Created by the bad job | Load ID equals the bad run | Re-embed its text with the correct encoder | | Overwritten by the bad job | Load ID equals the bad run, and the record existed before | Re-embed, or restore its previous vector if the text is unchanged | | Overwritten again later by a good job | Load ID is a later good run | Leave it; the good write already replaced it | | Deleted by the bad job, if it also deleted | The job's own log of deletes | Restore from the snapshot or re-ingest from source | The third row is the trap: a repair that blindly restores everything the bad job ever touched clobbers good writes made after it. ## Step 3: repair 1. Load last night's snapshot, or the stored-vector export, into a **scratch instance**, not into production. 2. For each record in the repair set, choose its good vector: re-embed the stored chunk text on the correct model, or copy the previous vector from the scratch instance when the text did not change. 3. Re-upsert those records with a new load ID, so the repair itself is traceable. 4. Remove the query-time filter. ## Step 4: verify - Count records still carrying the bad load ID: it should be zero. - Check vector properties on the repaired set: correct dimension, norms in the corpus range if vectors are normalised, no cluster of identical vectors. - Run a fixed set of known queries and compare top results with recorded expectations. - Re-embed a random sample and confirm stored and fresh vectors are near-identical. ## Prevent the next one - **Lineage on every record:** load ID, encoder version and write time, so one run can be found and undone. - **A validation gate before commit:** dimension, a norm check, a duplicate-vector rate and a small canary query set, failing the load before it reaches production. - **Staging:** write a load into a separate partition or collection, check it, then promote it. A store that keeps table versions can roll the whole table back to before the load, but that rolls back the good writes after it too, which is the same trade as a snapshot restore.
- The records carry no load ID. How do you find the bad 2%?Start from the job's time window if write times are stored, then test the vectors: wrong dimension, norms far from the corpus range when vectors are normalised, many identical vectors (often an embedded empty string), or a large distance between a stored vector and a fresh embedding of its text on a sample. Expect false positives, and add lineage fields before the next load.
- Why not restore the snapshot and replay the day's good writes from a log?That works if a durable, ordered log of every write and delete exists and replay is idempotent, and it is the right call when damage is wide. For 2% it is heavier: production goes offline or to a scratch copy, the replay must skip exactly the bad job's entries, and every delete since the snapshot must land. Surgical repair touches only damaged records.
- Restore last night's snapshot; losing a day of writes is acceptable.
- Re-running the same job unchanged will overwrite and fix the bad vectors.
- Corrupt vectors are obvious because searches will error on them.
- Every record the bad job touched should be restored from the snapshot.
- Lineage fields on records are overhead with no operational use.