A supplier's visual-inspection checkpoint passes a backdoor scan - what has that ruled out?

## What a backdoor is, and why detecting one is hard A backdoor is a **conditional trained into the weights**: on ordinary inputs the model behaves normally, and on inputs carrying a key chosen by whoever controlled training, it produces the behaviour that party wanted. The key fires at inference; the write access it required happened earlier, when the checkpoint was built. That is why ordinary evaluation does not surface it - a held-out set that contains no keyed inputs measures exactly the behaviour the adversary preserved. So the defender's problem is a search problem: somewhere in the space of all possible inputs there may be a pattern that flips the model's answer, and the defender must find it without being told what it looks like. That space is astronomically large. Nothing enumerates it. ## How published scanners make the search finite Every practical detection method buys tractability by assuming a **trigger family**. Three premises cover most of the published work: - **Reconstruction over classes.** For each output class, optimise the smallest input change that drives arbitrary inputs into that class, then compare across classes: a class reachable by an unusually small change is flagged. This assumes the key is a *small, additive, static, input-agnostic* pattern, and that only a minority of classes are keyed - the test is comparative. - **Statistical separation in the training data.** Look at internal representations of the training examples and search for a cluster that sits apart from its labelled class. This assumes you *have* the training set, and that poisoned examples separate in the representation the method inspects. - **Pruning or activation analysis.** Remove or study units that are rarely active on clean data, on the premise that a conditional is carried by dedicated capacity that clean inputs do not exercise. Each premise is a hypothesis space plus a compute budget - how many optimisation steps per class, how many classes, how large a patch, which clean inputs were used as the reference. Those numbers are the coverage of the scan. ## Why the assumption is the attack surface The party who publishes a checkpoint reads the same literature the defender does. That asymmetry is the leaf: the defence is fixed and public, the choice of key is made afterwards. An author who wants the artefact to pass can select a conditional whose shape violates whichever premise the expected scan rests on - a key that varies with the input rather than being one static pattern, a condition spread across the input instead of contained in a small patch, a behaviour keyed to a feature-level condition rather than a pixel pattern, or a model where more than one class is keyed so a comparative test has no clean baseline to be anomalous against. This is not free. Leaving the assumed family generally costs the adversary something - more control over training, more poisoned volume, a less reliable firing rate, or clean accuracy they cannot afford to lose because the artefact must be accepted on its merits. That cost is exactly what a scan buys you. It is a **price increase**, not a proof. ## Reading the result correctly The direction of the claim matters, and it is the thing interviewers listen for: | What the scan reports | What it actually establishes | | --- | --- | | No anomaly detected | No key of the searched family, at the budget spent, on the classes scanned | | Anomaly detected | A candidate of the searched family, subject to the tool's false-positive rate | | Nothing about behaviour | Whether the model is trustworthy on inputs you have not seen | A clean scan is a bounded negative. Reported as "the model is clean", it becomes an assurance claim that was never made by the measurement - and it is handed onward, into a review file or a risk register, where the bound quietly disappears. ## What to say in an interview Say that a scan is a search over an assumed trigger family, name the assumption in plain terms (small, static, identical across inputs, one target class), say that the publisher can read the same assumption, and finish with the honest formulation: the scan shifts residual risk and prices out careless adversaries; it does not clear the artefact. Then say what you would rely on instead - behavioural acceptance testing on inputs you collected yourself, and limiting what a single model decision is allowed to do without a second check.