In an adversarial ML evaluation, what does it mean to grant an attacker white-box access?

## The word names an assumption, not an event In adversarial machine learning, *white-box* and *black-box* describe **what the adversary is assumed to have**, not something that happened to you. A white-box adversary is assumed to hold: - the **trained weights** of the model; - its **architecture** — layer structure, input preprocessing, output head; - the ability to compute **gradients of a loss with respect to the input**, because holding weights and architecture means you can differentiate the function end to end. A black-box adversary holds none of that and interacts only through predictions. In between, an adversary might know the architecture and the rough data distribution but not the parameters. That intermediate case is a different assumption with different consequences and is not what white-box means. ## Why a lab is *handed* the weights Consider a regulated setting: a chest-radiograph triage classifier that flags studies for urgent read, submitted by a device vendor to an accredited evaluation lab as a condition of clearance. The submission package includes the weights, the architecture, the training recipe and a differentiable copy of the model, transferred under NDA, and the contract explicitly forbids the lab from pretending it is an outside tester. That looks backwards until you ask what the alternative measures. If the lab is only allowed to send inputs and read outputs, and it reports that it could not make the model misread a study, the report has two possible causes and no way to separate them: 1. the model genuinely resists perturbed inputs; or 2. the lab could not find the perturbation because it did not have the weights. Cause (2) is a statement about the lab's information, not about the device. Clearance cannot rest on it, because the vendor's secrecy is not a safety property — weights leak, models get shipped to devices, and staff move. Granting access deletes cause (2) from the report. What comes back is then attributable to the model. ## The result is a bound on everybody else Because the white-box adversary is the most capable one you can define for that model — anything a weaker adversary learns, this one already had — the attack success measured under it is an **upper bound on what any weaker adversary can achieve**, and the surviving accuracy is correspondingly a **lower bound** on how the model fares against them. That is the whole reason the stance is worth its cost: one expensive evaluation constrains a whole family of cheaper adversaries you will never enumerate. The inverse is also true and is the practical reason results are unreadable without the assumption stated. A reported attack success rate with no access assumption attached could have come from an adversary holding gradients or from one holding a single returned label, and those two numbers differ by an order of magnitude on the same model. This is why every credible robustness claim states its access class alongside its perturbation family and radius. ## What granting access is *not* - **It is not a claim that the weights were stolen.** No breach is being asserted or predicted. The grant is a modelling choice made before any testing begins. - **It is not an assumption about the training data.** White-box concerns the trained artefact. An adversary who additionally holds the training set is a further, separate assumption, and privacy attacks turn on that distinction. - **It is not the software-testing sense.** In QA, white-box testing means test cases informed by internal source and control flow. The words collide; the objects do not. - **It is not 'black-box model' in the interpretability sense.** There, black-box means a model whose reasoning is opaque to a human. That is a property of the model. Here, black-box is a property of the attacker's access. Because three fields use these two words for three different things, a careful engineer says *access* out loud: 'white-box access — weights and gradients assumed available'. ## What withholding weights is actually worth Keeping parameters private is a real and worthwhile cost control: it makes an adversary pay in queries, in money, or in the effort of building a stand-in that agrees with your model where it matters. It is not a boundary, though, and it does not change what the model does when someone eventually holds it. So the operating posture is to keep weights private *and* to evaluate as though they were not, and to report both numbers with their assumptions attached rather than one number with none.