A user deletes a confidential document, but its chunk still appears in assistant answers minutes later: what can cause it, and how do you tell which?
answer
- start from the cited chunk ID
- intermittent means one replica
- did the delete match anything
- caches and chat history
- exclude at query time meanwhile
basics
~20 sThe delete may still be applying, a lagging replica may still serve the chunk, the delete may have matched no stored record, or the text may come from a cache or chat history. Look up the cited chunk's ID on each replica to tell which.
solid answer
~50 sI would start from the retrieval trace of one bad answer to get the chunk ID, then do exact ID lookups against each replica. Present on some replicas only, with the chunk appearing intermittently, points to a **lagging replica**. Present everywhere seconds after the acknowledgement usually means the delete is **still in the apply queue**; present everywhere long after means it probably **matched nothing**, so I compare its target with the stored ID. Absent everywhere means the text comes from **outside the store**: an answer cache, the conversation history, or another collection behind an alias. Compaction is not a cause on the usual tombstone design, because tombstoned records are filtered from results before compaction. Meanwhile I filter the document out at query time on every search, purge caches, and retire that filter only after a per-replica check passes.
go deeper
Recall that an acknowledged delete can still be pending on some stores, and that caches and chat histories can repeat text the store no longer holds.
Explain how tombstones hide records before compaction, and why an intermittent hit points at one replica rather than at the whole store.
Drive the triage from a cited chunk ID with exact per-replica lookups, separate the four causes, and bridge the window with a query-time exclusion.
Treat it as an exposure with a window to report: decide which deletes must wait for apply, and who owns caches and aliases that outlive a delete.
## Start from the evidence, not the store A deleted confidential document whose chunk still turns up in answers is a stale-data incident, and in a security review it is a finding. Before touching the store, get the **retrieval trace** for one bad answer: which chunk IDs the search returned, from which collection or index, served by which replica if the store reports it, and when. Without a chunk ID you are guessing; with one, every cause below can be tested in minutes. ## The four families of cause 1. **The delete is still applying.** On a store that applies writes asynchronously, an acknowledged delete waits in the same apply queue as inserts. Behind a bulk load the queue can be long, so the chunk stays visible until its delete is applied. 2. **A replica is behind.** The delete has applied on some replicas but not on one that still serves queries. The signature is intermittency: on three replicas behind a load balancer, roughly one search in three returns the chunk. 3. **The delete matched nothing.** The request named IDs, a filter or a collection that does not match what is stored, so the store correctly deleted zero records. Deciding which chunks a source change should delete is the ingestion pipeline's job; here you only check whether the request named the record that is actually stored. 4. **The text is not coming from this store.** An answer cache or semantic cache replays an earlier response, a conversation history already quotes the chunk, or a second copy still holds it: an older index behind an alias, a staging collection, or a store restored from a snapshot taken before the delete (restores that revive deletes are a recovery subject of their own). A common wrong answer is a fifth: "compaction has not run yet." On the usual design a delete writes a **tombstone** or sets a bit in a deletion bitmap, and every search filters marked records out. Compaction later removes the bytes. A record waiting for compaction is hidden, so compaction delay matters for erasure, not for what users see. The other wrong answer is "approximate search randomly resurfaces deleted vectors": approximation loses results, it does not return records that every replica has removed. ## A triage table | Observation | Check | Likely cause | |---|---|---| | Chunk returned intermittently | ID lookup sent to each replica directly | a lagging replica | | Present on every replica, delete acknowledged seconds ago | apply backlog or queue depth | delete still applying | | Present on every replica long after the acknowledgement | compare the delete's target with the stored ID and filter fields; check the delete's reported match count | delete matched nothing | | Absent on every replica, text still in answers | caches, conversation history, other collections and aliases | a copy outside this store | Run the checks with **exact** reads - ID lookups and filter counts - rather than a similarity search, because an approximate search can miss a record that is still there and make a live chunk look deleted. ## Bridging the window while you fix it - Put the deleted document's ID on a **query-time exclusion list** that every search applies as a filter. It hides the chunk on every replica immediately, whatever the cause inside the store. - **Purge caches** whose entries were built from the affected chunks, and drop the chunk from active conversation contexts where the product allows it. - For deletes that must not lag, such as confidential or revoked content, send them with the store's wait-for-apply option or a stronger write consistency, and verify with a per-replica read-back before telling anyone the document is gone. - Retire the exclusion only after that read-back passes on every replica. ## Why the diagnosis matters The fix differs by cause: a lagging replica needs health and capacity work, a pending apply needs the delete path prioritised or waited on, a delete that matched nothing usually points at how IDs, filters or collection names are derived upstream, and a copy outside the store needs cache and alias hygiene. Re-sending the delete until the symptom stops fixes none of them and hides the evidence of which one it was. Record the timeline - delete acknowledged, chunk last seen, each replica confirmed clean - because a confidential-data exposure is reported with its window.
- The chunk appears in about one search in three on a three-replica store. What does that tell you?That one replica has not applied the delete while the other two have, and the load balancer spreads queries across them. Confirm with an ID lookup sent directly to each replica, check that replica's apply backlog and health, and until it catches up, exclude the document at query time or take the replica out of rotation.
- Why isn't 'compaction has not run yet' an explanation for the chunk appearing in results?On the usual design a delete writes a tombstone or marks the record in a deletion bitmap, and every search filters marked records out; compaction only removes the bytes later. A record waiting for compaction is hidden from results, so compaction delay matters for an erasure request, not for what users see.
- How do you stop the exposure while the cause is still being found?Add the document's ID to a query-time exclusion list that every search applies as a filter, which hides it on all replicas at once. Purge caches built from its chunks and remove it from live conversation contexts where possible. Keep the exclusion until a per-replica read-back shows the delete applied everywhere.
- Approximate search randomly resurfaces deleted vectors, so this is expected noise.
- The chunk shows up because compaction has not physically removed it yet.
- The delete call succeeded, so the vector store cannot be the source.
- Just re-send the delete until the chunk stops appearing.
- Text in an answer must come from this search, never from a cache or chat history.