In an adversarial-ML evaluation stack, what is the difference between an attack library (such as the Adversarial Robustness Toolbox or Foolbox) and a command-line harness that drives one (such as Counterfit)? Which of the two decides what attacks are available to you and how far you can tune them?

Get this separation right early, because most later confusion on an adversarial-robustness engagement — whose number is it, why did the attack "fail", why did the bill arrive — traces back to collapsing the two layers into one word: "the tool". ## The library layer, at the level of the object An **attack library** is a Python package you import into your own process. Take the Adversarial Robustness Toolbox (ART) as the worked example. 1. You first wrap the thing under test in an *estimator*: `art.estimators.classification.PyTorchClassifier(model=..., loss=..., input_shape=..., nb_classes=..., clip_values=(0, 1))`. The estimator is the adapter — it teaches the library how to get predictions out of your model and, for gradient attacks, how to get gradients. 2. Then you construct an *attack object*: `art.attacks.evasion.ProjectedGradientDescent(estimator=clf, eps=0.03, eps_step=0.007, max_iter=40, num_random_init=5)`, and call `attack.generate(x=x_test)`. Foolbox has the same shape under different names — `foolbox.PyTorchModel(model, bounds=(0, 1))` plus `foolbox.attacks.LinfPGD()` called with an `epsilons=` list. Every knob that decides whether the attack is *strong* is a constructor argument on a library object: - `eps` (how much perturbation is permitted), - `eps_step` (how far one iteration moves), - `max_iter` (how long the search runs), - `num_random_init` (how many restarts), - and the estimator's `clip_values` (the valid input range). ## The harness layer A **harness** is a driver over those objects. Counterfit is the canonical command-line example: an interactive shell in which - `list targets` shows the target classes you registered, - `interact <target>` selects one, - `list attacks` shows the attacks it re-exports from the libraries beneath it, - `use <attack>` picks one, - `show options` prints the subset of that attack's arguments it surfaces, - `set <param> <value>` changes them, - `run` executes, and `show results` prints a table. It contains no optimisation code of its own. What it adds is a uniform target description, a loop over a menu, failure handling partway through a sweep, and a comparable result record — roughly a day of driver work you skip, and a day you would otherwise re-spend on every new project. ## What a run costs, and who sets that The cost lives in the library's parameters no matter who invokes them. A white-box PGD at `max_iter=40` with five restarts is about 200 forward-and-backward passes per input: seconds on a GPU, minutes for a few hundred inputs. A black-box decision-based attack is four orders of magnitude away — ART's `HopSkipJump` defaults to `max_iter=50` with `max_eval=10000`, i.e. tens of thousands of *model calls for one single input*. Point that at a metered hosted endpoint from a harness that offers one convenient `run` command, with a hundred test inputs, and you have committed to a seven-figure call count and a real invoice. The harness's virtue — one uniform interface — is exactly what hides the fact that two adjacent menu entries differ in cost by 10,000x. ## Where the number misleads - The classic is a **unit mismatch** that the harness makes invisible. ART's `eps=0.3` default is written for images normalised to [0, 1] — the MNIST convention. Register a target whose inputs are 0-255 pixels, or unnormalised tabular features, leave `eps` at the default because `show options` never surfaced it, and the attack is searching inside a ball too small to change the input at all. Every row of `show results` reads "failed", and the tempting reading — "the model is robust" — is wrong; the true reading is "the units were wrong". - The second misreading is **coverage**: "we ran everything the tool listed" has as its denominator *attacks the harness exposes*, not attacks that exist for that model type. Both errors report the harness's limits as the model's strength. ## What to check - Run `pip freeze` inside the harness's environment to learn which library and version actually produced the result — that names the real owner of the number. - Diff `show options` against the library attack's constructor signature to see what is fixed silently. - Confirm `clip_values`/`bounds` and `eps` are in your model's real input units by running one attack at a deliberately huge `eps` and verifying it succeeds; if it cannot win when handed an absurd allowance, the plumbing is broken. - And check you can still `import` the library directly in the same environment, because that is your escape hatch when the menu runs out.