A filed RAG-poisoning finding will not reproduce on demand - how do you tell a build-window artefact from a dead finding?
answer
- you are testing a different index than the reporter did
- retrying this afternoon samples one generation
- reconstruct the timeline of writes and builds
- check source and index as separate artefacts
- one success is one draw, not a rate
basics
~20 sCompare clocks, not attempts. Repeated queries inside one index generation settle nothing; establish when the document was written, which builds have run since, whether the source still holds it, and whether the current index contains a chunk from it.
solid answer
~50 sRetrying the same query all afternoon samples a single index generation, so it cannot separate "never worked" from "worked against a build that no longer exists". The reproduction has to cross a build boundary. Reconstruct the timeline: when the document landed on the source, which builds ran between then and the reporter's observation, whether anyone has since removed or re-curated it, and whether a chunk from it is in the index right now. Then check the index and the source as two separate artefacts, because they disagree for a window in both directions. A negative result today proves only that the current index returned no matching chunk for the queries tried - it does not prove the write failed or the report was wrong. Equally, one success proves the construction worked once, against one build.
go deeper
Know that answers over a scheduled corpus depend on when the index was last built, so an attempt today may be testing a different corpus state than the one a report describes.
Explain why repeating a query within one index generation adds no information, and what a rebuild can change: membership, chunk boundaries and the vector positions of everything in the corpus.
Show a triage method built on a timeline and on reading the source and the index separately, and state each verdict in terms of what the evidence supports rather than what the retries felt like.
Own the standard for what this class of report may claim: distinguish a live effect, a historically evidenced one with a closed window, and an unsupported one, and refuse verdicts the method could not reach.
## Why this class of finding resists reproduction A report arrives: a document written into a corpus source caused an assistant to answer from a planted passage. The engineer who picks it up cannot make it happen again. Before concluding anything, it is worth naming why this particular class is unusually slippery. The effect depends on the state of an **index** that is regenerated from **sources** on a schedule the reporter did not control and probably did not record. Between the original observation and the reproduction attempt, that index may have been rebuilt any number of times. A rebuild can add the document, drop it, re-chunk it differently, or re-embed the whole corpus. So the system under test is not the same system the reporter tested, and no amount of repetition this afternoon changes that - repetition inside one index generation is repetition of one draw. ## The timeline is the evidence The useful reconstruction is a sequence, not a test: 1. When did the document land on the source? 2. Which builds ran between that write and the reporter's observation? 3. Did the reporter's query actually retrieve a chunk from that document, or only produce a plausible-looking answer? 4. Has the source been touched since - removed, re-curated, moved out of the crawl's scope? 5. Does the current index contain a chunk from that document at all? 6. Which query was used, and was it the query the passage was written to win? Questions 1, 2 and 4 are about clocks; 3, 5 and 6 are about artefacts. Most failed triage skips straight to "run the query again", which is the one action that answers none of them. ## Source and index are two different artefacts The central discipline is refusing to treat them as one. The index reflects the sources as of the last build. So during any window: - a document present on the source may be absent from the index, because no build has run since it was written; and - a document absent from the source may be present in the index, because no build has run since it was deleted. An engineer who checks only the share can conclude "it is gone, so the finding is dead" while the assistant is still quoting it. An engineer who checks only the assistant's answer can conclude "it never worked" while the chunk is queued for tonight. Both readings are ordinary and both are wrong. ## What each result is allowed to establish | Result | Supports | Does not support | |---|---|---| | Five queries return nothing | the current index returned no matching chunk for those queries | that the document was never ingested | | The document is not on the share | it was removed from the source | that the index no longer holds a chunk from it | | One query returns the planted passage | it worked once, against this build | that it works reliably or across rebuilds | | The corpus was re-embedded last week | ranking may have shifted | that the document was dropped | The last row is the one that catches people: a re-embedding of the whole corpus changes every vector's position, so a passage that used to win a query may lose it without anyone touching the document. ## Adjudicating it Once the timeline is reconstructed, three honest verdicts are available, and they are genuinely different: - **Confirmed, schedule-bound.** A chunk from the document is in the current index and can be retrieved for the query it was written for. The intermittency was a build boundary, not luck. - **Confirmed historically, no longer live.** The evidence shows it worked against an earlier build and the document has since been removed or re-curated. That is still a real finding about the pipeline, with a closed window; saying so is not the same as calling it invalid. - **Unsupported.** No chunk from the document is or was in the index, and nothing in the timeline explains an effect. Then the reported answer needs a different explanation - the model may simply have produced a plausible statement without retrieving anything, which is a different problem. What is not available is "could not reproduce, closing", written after an afternoon of retries. That verdict claims to have measured something the method could not reach. The honest report says which index generation was tested, what the source and index each contained at that moment, and which of the three verdicts the evidence supports - and if the evidence only supports one draw, it says the finding rests on one draw.
- The document is gone from the share and the assistant still quotes it. What do you write in the report?That the source no longer holds it while the current index still does, and that the two will agree again after the next build. Both facts are true at once and the report is wrong if it states only one. It also means any claim that removal ended the exposure is about the share, not about what retrieval can return.
- Is a finding that reproduced once, months ago, against a since-rebuilt index still a finding?Yes, but about the pipeline rather than about the current answer. It evidences that a write into that source reached the index and was retrievable, which is a property of the ingestion path and survives the particular document. What it does not support is any claim about today's index, and conflating the two is how these reports lose credibility.
- What is the risk of asking the model whether it saw the document?Its answer is not a retrieval log. A model can describe sources it did not receive and omit ones it did, so its self-report is evidence about generated text, not about what the retriever returned. The retrieval result is the artefact worth reading.
saying these in an interview costs you the question
- Closes the finding after retrying the same query all afternoon
- Treats one successful reproduction as a reliability claim
- Checks only the source share, or only the answer, never both
- Assumes a negative retrieval proves the write never landed
- Ignores that a corpus re-embedding shifts ranking without touching documents
- Accepts the model's self-report as evidence of what was retrieved