How much assurance can you claim from a backdoor scanner a year after its method went public?

## The decay is directional, and that is the whole judgment A detection method's assumptions become public the day it is published. From then on, any party choosing a key for a checkpoint they intend to hand over can read them. That is not a defect of the tool and it is not fixed by a newer tool, which will also be published. So the honest model of a scanner's value over time is not "it degrades" but **it degrades against one class of adversary and not against the others**: | Who you are facing | What a public scan is still worth | | --- | --- | | A supplier who never thought about this | Nearly its original value - cheap keys are found | | A reused or copied artefact from elsewhere | High - off-the-shelf conditionals sit inside the searched family | | An accidental correlation learned from data | Some - it can surface a class reachable too easily | | A supplier who read the method and is targeting you | Little, and it falls further each quarter | The uncomfortable consequence, and the sentence worth saying out loud in a principal interview: **the assurance is weakest exactly where the artefact is worth attacking.** A checkpoint on a low-stakes internal task is well covered; the one running a high-value production decision is the one an adversary would bother to design against. ## The claim you let leave the team The deliverable of a review is a sentence somebody else will act on, often months later and without the context. The failure mode this leaf exists to prevent is a bounded negative turning into an assurance claim on the way out of the room: "scanned clean" becomes "verified", and by the time it reaches a risk register the family, the cap, the class coverage and the budget are gone. So set the wording as policy, not as a habit: - what goes in the file is the scoped negative, with the family and the budget attached; - "no backdoor" is not a conclusion any scan can produce, and nobody may write it; - the compensating control and its rate are recorded next to the negative, because the negative alone is not an acceptance decision. This is genuinely an organisational call rather than a technical one: you are deciding what your team is permitted to assert, and you will be arguing it with someone who wants a yes. ## Where the next unit of budget goes Given that the scan's coverage is fixed and public, additional spend on the same axis buys little. The controls that keep working are the ones that never needed to know the key: - **Acceptance evaluation on data you own** - collected under your conditions, labelled by you, weighted toward the decisions that hurt when they are wrong. - **A sampled independent re-check in production**, sized by consequence rather than by convenience, and biased toward the direction of loss. On a pass/fail line, sampling the passes is what matters. - **Monitoring the decision distribution** by source, batch, and period. A conditional that fires in the field moves the mix even when nobody can name the key. - **Reducing what one verdict does alone.** If a model decision has an independent check behind it, a hidden conditional degrades a metric instead of causing the loss directly. - **Not accepting the artefact** for the small set of decisions where none of the above is affordable - training on data you control, or keeping a person in the loop. ## Two traps in the periodic-review version of this question First, **re-running the same tool on the same unchanged artefact next quarter adds nothing**. It produces the same bounded negative. What would add something is a method whose searched family is genuinely different, which is a procurement question, not a scheduling one. Second, **stacking tools is not corroboration**. Two published methods that both assume a small, static, local key overlap heavily, so the union of coverage grows far less than the count of tools suggests. Report families cleared, not tools run. ## The answer to the stakeholder who wants a yes or no Give a scoped yes and the control that covers the rest: no key of the searched family was found, at this budget, across these classes, against reference data we supplied; the residual is a conditional of a different shape, and it is covered by an acceptance set we own and a sampled re-check at a stated rate. Saying that a scan cannot answer "is it backdoored" is not evasion - it is the accurate scope of the measurement, and owning that distinction is what the level is being tested on.