How does an ingestion pipeline prove a delete landed on every replica of a vector store, rather than merely that it was acknowledged?
answer
- acknowledgement is only a claim
- every replica, not the endpoint
- exact reads beat similarity search
- deadline, alert, keep the filter
- record a receipt per deletion
basics
~20 sRead it back: after the acknowledgement, check each replica with exact ID lookups and a document-filter count, poll until every one reports zero, and record when. An approximate search that no longer returns the chunk proves little on its own.
solid answer
~40 sThe acknowledgement says the store accepted the delete; proof means reading the store back the way queries reach it. I would address each replica directly, or use a read level that consults all of them, and run **exact** checks: look up every deleted chunk ID and count records matching the document's filter, expecting zero. Absence from an approximate search is weak evidence because its recall is below one. The job polls until every replica is clean, or waits until each replica's applied position passes the delete where the store exposes that, and records a per-replica confirmation time. Past a deadline it alerts and keeps the query-time exclusion filter in place. A passed proof shows the record is hidden everywhere, not that its bytes or cached copies are gone.
code
pseudocode · 18 linesfunction prove_delete(doc_id, chunk_ids, deadline):
pending = set(all_replicas())
while true:
for replica in copy(pending):
if not replica.reachable():
continue # silence is not evidence
found = replica.lookup_by_id(chunk_ids) # exact, not similarity search
left = replica.count(filter: doc_id == doc_id)
if found is empty and left == 0:
record_confirmation(doc_id, replica, now())
pending.remove(replica)
if pending is empty:
retire_query_time_exclusion(doc_id)
return PROVEN
if now() > deadline:
raise_alert("delete not visible everywhere", doc_id, pending)
return UNPROVEN # exclusion stays in place
sleep(poll_interval)go deeper
Recall that an acknowledged delete is a claim that the store accepted it, and that proving it means reading the store back.
Explain why the check must reach every replica and use exact lookups, since an approximate search can miss a record that is still stored.
Design the loop: per-replica exact reads or applied positions, a deadline that alerts, a receipt, and an exclusion filter retired only on proof.
Decide which deletions need proof and receipts, and state plainly what a proof covers and what compaction, retention and caches leave open.
## An acknowledgement is a claim; a read-back is evidence When a vector store acknowledges a delete, it tells you the request was accepted, and depending on the store and its settings, that the delete is logged or applied somewhere. It does not tell you that every replica serving searches has applied it. For content that must be gone - a revoked document, a confidential file, an erasure request - the pipeline needs **evidence**, and evidence comes from reading the store back, the same way a user's query would reach it. ## What a sound proof checks 1. **Every replica, not the endpoint.** A load-balanced endpoint can send each check to the same fresh replica and pass while another still serves the chunk. Address replicas directly, or use a read level the store documents as consulting all of them, or read each replica's applied-operation position where the store exposes one. 2. **Exact reads, not similarity search.** Look up each deleted chunk ID, and count records matching the document's filter, expecting zero. An **approximate nearest-neighbour** search has recall below one, worst under restrictive filters, so "the search no longer returns it" is weak evidence: a record that is still stored can simply fail to surface. 3. **The path users hit.** On some stores the lookup path and the search path read different structures. Where that is true, add a probe: search with the deleted chunk's own vector, kept in the deletion job's record, filtered to the document, and expect no deleted IDs back. It complements the exact checks; it never replaces them. 4. **A position, where available.** If the store reports an operation sequence number for the delete and each replica's applied position, waiting until every replica is past it is cleaner than polling for absence. 5. **A deadline and a loud failure.** A replica that stays stale past the deadline is an incident, not a retry loop. ## The proof loop The pseudocode below keeps a set of replicas still to confirm, removes each one that reports zero matches, and either retires the query-time exclusion when the set is empty or alerts and leaves the exclusion in place when the deadline passes. An unreachable replica stays pending: silence is not evidence. Two parameters need thought: - **The deadline** should come from the measured visibility window, for example a high percentile of past deletes' acknowledgement-to-last-replica time with a margin, so a normal delete passes and a stuck replica fails quickly. - **The poll interval** trades load against detection time. Exact lookups by ID and a filtered count are cheap, but thousands of deletes polled every second add up, so batch the checks per replica. Replica membership matters too: a replica added while the job runs should be checked before the proof passes, since it may have been built from a copy taken before the delete. ## What the receipt records A proof is only useful if it can be shown later. Record, per deletion: - the document ID and the chunk IDs the delete targeted, and the store's reported match count; - when the delete was acknowledged; - per replica, when it first read back clean and by which check; - the time the query-time exclusion was retired; - anything the proof does not cover and who owns it. The interval from acknowledgement to the last replica's confirmation is the delete's real **visibility window**. Tracked over time it tells you whether that window is growing, for instance during bulk loads. ## What the proof does not prove | Claim | Proven by the per-replica read-back? | |---|---| | No search on any live replica returns the record | yes | | The record's bytes are gone from segment files | no; that waits for compaction | | Older versions, logs or snapshots no longer hold it | no; that is retention and backup policy | | Caches and chat histories no longer repeat it | no; those live outside the store | Keeping these apart matters when someone asks whether a document "is deleted": the honest answer names which of these rows has evidence behind it. ## Bridging until the proof lands Between the acknowledgement and the last confirmation, a **query-time exclusion list** of deleted document IDs, applied as a filter on every search, keeps the chunk out of results on every replica. The proof loop is what entitles the pipeline to remove that filter; without the loop, the filter either stays forever and grows, or is removed on hope.
- Can the proof work without addressing replicas directly?Yes, if the store offers a read level documented to consult every replica, or exposes each replica's applied-operation position so you can wait until all are past the delete. Without either, checks through a balanced endpoint can keep hitting a fresh replica and pass while another still serves the chunk.
- What does a passed proof still not cover?It shows no live replica returns the record. It says nothing about bytes still in segment files before compaction, older versions or snapshots that hold the record, caches outside the store, or chat histories that already quoted the chunk. Each needs its own owner and its own evidence.
- A success status on the delete call proves the vector is gone.
- If a similarity search no longer finds it, the delete has landed.
- Checking the leader is enough, because replicas always copy it eventually.
- A passed delete proof also shows the bytes are gone from disk.
- Re-sending the delete until it succeeds is the same as verifying it.