You lead ML security for several product teams and must standardise how adversarial-robustness evaluations are run. How do you decide between adopting a third-party attack harness, building a thin in-house driver over the attack libraries, or letting each team script directly — and how do you keep that decision reversible?

This is a platform decision wearing a tooling costume, and strong answers treat it that way. The question is not "which harness"; it is "what must be identical across teams for their numbers to sit in one table". **Frame it as: what must be common?** Three artefacts, none of which is a tool. 1. *The target contract* — how a team describes the thing being evaluated: the model interface, the access mode (white-box gradients, score-only, decision-only), the preprocessing applied before the model sees an input, and the input range the perturbation budget is expressed in. Without a shared contract, an `eps` from team A and an `eps` from team B are different physical quantities. 2. *The result schema* — what a run records: library and version, attack, every setting, the query or compute budget, the access assumption, the denominator, the success criterion, and the outcome. This artefact outlives every tool change. 3. *The reporting bar* — what an evaluation must contain before a claim may be drawn from it: a swept budget rather than a single point, a sanity check against an undefended reference model, and full provenance. Standardise those and the harness becomes an implementation detail rather than the contract. **Then choose the layer.** *Adopt* a third-party harness when its covered input domains and access modes match most of the estate, its pinned stack can live in an isolated image, and its surfaced parameters suffice for triage — adoption is near-free and the maintenance is someone else's, provided upstream is still alive; check the commit history, because a dormant wrapper hands you the maintenance anyway, later and by surprise. *Build a thin in-house driver* when you need parameter reach the harness does not offer, when the estate is unusual, or when the pins cannot be made to fit; keep it genuinely thin — it calls attack libraries, it does not reimplement attacks — and let it die when a third-party option becomes adequate. Budget it honestly: a thin driver is a couple of engineer-weeks to write and a standing tax of a few days a quarter chasing upstream API changes. *Let teams script directly* only while there are one or two teams and no cross-team claim is being made; it costs comparability from the first week. The usual good answer is a hybrid: harness for breadth-first sweeps and routine regression, direct library calls for anything that becomes a published claim. **Where the number misleads at this altitude.** A cross-team robustness dashboard is the most dangerous artefact this decision produces, because it makes non-comparable numbers look ranked. A team can appear robust because its harness surfaced fewer parameters, ran a smaller attack menu, used the all-inputs denominator rather than originally-correct, tested at a smaller budget, or hit a query cap against a metered endpoint and truncated the sweep. Every one of those reads on the tile as strength. Before any ranking is published, the dashboard must carry the budget, the denominator, the access mode and the attack set per row — and a row that cannot state them shows as "not measured", not as a good score. The corollary: never let "the tool doesn't support that" quietly become "that risk is fine". Record it as unmeasured. **Keeping it reversible.** Reversibility is engineered, not hoped for. Own the schema and make every tool a producer that fills it. Keep evaluation environments disposable and versioned with results, so a tool change is a new image benchmarked against the old rather than a mutation. Retain raw artefacts — original inputs, perturbed inputs, model decisions — beside the summaries, because those are what a future tool can re-score; a rendered summary table cannot be re-scored by anything. And keep the tool's bespoke vocabulary out of your reporting language: every leaked concept is a chain to that tool. **How to tell you chose wrong.** Teams routing around the standard with private scripts; quarter-over-quarter numbers that cannot be compared because the stack moved and nobody froze it; and the phrase "our tool doesn't support that" appearing as the reason a risk went unmeasured rather than as an entry on the risk register.