An overnight Counterfit scan that had run fine all week burned a month of endpoint quota after a colleague switched on the optional search over attack parameters. Where does that multiplier come from, and how would you bound the next run?

**What the toggle changes.** In Counterfit's command loop, the optional search over attack arguments sits next to `run_attack` and looks like one more setting. It is not a setting; it changes the class of the job. Without it, an attack runs once with the arguments you set and the cost is fixed: seeds × iterations × calls per iteration. With it, the attack's arguments become the search space of a hyper-parameter study (an Optuna study in the shipped implementation), and **one trial is one complete execution of the attack over the seed set**. The study's objective — attack success rate, or success at a minimal perturbation — is a single number that can only be computed by running the whole attack to the end. Nothing is amortised between trials: no predictions are cached, no seeds are pruned, no partial run is reused. So the battery's cost picks up a plain multiplier: ``` before: attacks × seeds × iterations × calls_per_iter after: attacks × TRIALS × seeds × iterations × calls_per_iter ``` Ten trials over three attacks is thirty complete attacks against a metered endpoint. And the multiplier is worse than uniform, because a search that is working spends its later trials in the region where the objective looks promising — which is normally the argument sets that push the attack to run its full iteration budget rather than exiting early. The trials that succeed are the expensive ones. **Work it out loud in the answer.** Three attacks, twenty seeds, one attack measured at about five thousand queries over that seed set: fifteen thousand queries a night for the battery. Turn on ten trials of search for each attack and you are at a hundred and fifty thousand. At a tenth of a cent per call that is a rounding error; at two cents a call it is three thousand dollars; at 200 ms serialised it is over eight hours of pure wall clock before any of it overlaps. The endpoint did not change and the attacks did not change — the shape of the job did. **Diagnosing it after the fact.** Pull the **per-attack** query counters, not the elapsed time. A search that burned its entire budget inside one attack and barely touched the others looks identical, in wall clock, to an even spread — and the fix is completely different. Then check three things: whether trials can abandon a hopeless configuration early or always run to completion (a study with no pruning pays full price for every dead end); whether the search ran over the full seed set, because `len(X)` multiplies inside every trial; and whether the trial count was one you chose or a default nobody read. **Where the number misleads.** The tuned result is the trap. A search reports the argument set with the best objective *on the seeds it searched over*, and with eight or twenty seeds that is a small sample being optimised against directly. The winning success rate is therefore fitted, and quoting it as the model's exposure overstates it — sometimes badly. The honest figure is the winning arguments re-measured on seeds the search never saw. The mirror error is just as common in the other direction: reading a *failed untuned* attack as evidence of robustness. A default step size or epsilon is not a serious adversary's choice, so an untuned failure understates exposure. Both readings come from treating one run's number as a property of the model rather than a property of a configuration. **How I would bound the next run.** - **Cap the trials explicitly**, written next to the attack list and reviewed like any other budget number, rather than accepting whatever the default is. - **Two-phase it.** Search cheaply — on a small seed subset, or against a local or surrogate copy of the model where queries are free — then send only the winning arguments to the metered endpoint for one confirmation run. The multiplier lives entirely in the search phase, so move that phase off the meter. - **Hard-cap queries in the target itself**, so the worst case is a raised exception at a threshold you chose rather than a discovery on the bill. - **Report on held-out seeds**, with the trial count and the search space stated next to the score. **The judgment point.** The failure here was not enabling the search. Tuning is legitimate and necessary: untuned attack arguments understate a model's exposure, and a defender who reads that as safety is being misled by your report. The failure was enabling an open-ended optimisation against a metered production endpoint with no trial cap, no query cap, and no named owner of the budget it was about to spend.