After fine-tuning an inherited checkpoint, a disclosed backdoor's success rate fell from 96% to 11% - what does that establish?

## The result is real; the claim people make from it is not A measured drop from 96% to 11% is genuine evidence and worth having. The failure mode is the sentence that gets written underneath it: *the fine-tune removed the backdoor*. Everything useful in this question is in the gap between the measurement and that sentence. ## What the number is actually attached to Four scopes, all narrow: 1. **One key.** Somebody disclosed a key for this checkpoint family, which is the only reason you could measure anything. A conditional is fired by a feature the adversary chose; a key nobody published was never in your test set. Testing what you were told about tells you about what you were told about. 2. **One recipe.** A step count, a corpus, a learning rate, a freeze policy, and in this table a pruning stage. Change any of them and the residual rate changes. Another team running 'the same' full fine-tune on a different corpus can land somewhere else entirely. 3. **One input population.** A success rate is measured over inputs you assembled and keyed. A different population - longer inputs, a different domain slice, a slightly varied key - gives a different rate, and the adversary is not obliged to use yours. 4. **One direction.** The measurement establishes that this behaviour got harder under this treatment. It bounds nothing. There is no guarantee here, only an empirical rate. ## Why 11% is not the comfortable number it looks like Robust-accuracy intuition from average-case performance does not transfer, because this is not an average case. The adversary chooses the input, chooses when to send it, and can send many. A conditional that fires roughly one time in nine, at their discretion, on an input they construct, may be entirely sufficient for the payoff - especially where the payoff is a single retrievable document surfacing in an internal search index, or a single decision going their way. Ask what one successful firing is worth before deciding whether a rate is small. ## Which row is doing the work In a table shaped like the one above, the frozen-body row barely moves (93.8%), the full fine-tune roughly halves it, and the large drop arrives only with pruning. That matters for two reasons. First, if your production pipeline does not prune, the row you should be reading is 41.5%, not 11.3% - people quote the best row in the table and ship a different recipe. Second, the pruning row has its own bill: clean accuracy slipped, and on a per-slice basis a compression step usually costs rare classes and small subgroups more than it costs the average. You are trading measured task quality for an unquantified reduction in an unquantified risk, which is a decision, not a control. ## The reporting columns you would ask for If you were handed this table in a review, the questions are mechanical: which keys were tested and who disclosed them; how many inputs the rate was measured over; whether the deployed pipeline matches the row being quoted; whether clean accuracy is reported per slice as well as in aggregate; and whether anyone re-measured after the final production adaptation rather than a lab approximation of it. A row missing any of those is a number without a scope. ## The honest write-up Something close to: *under our production adaptation, the one publicly disclosed key for this checkpoint family fired on 11% of the keyed inputs we constructed, down from 96% on the published weights; this result covers that key only, at that recipe, and we have no test for undisclosed keys.* That sentence is defensible in front of anyone. 'The backdoor was removed' is not, and it is the sentence a reviewer is listening for.