Fields the Endpoint Returns

## What a threat model has to state A threat model for a deployed model has three parts: what the adversary is trying to achieve, how much they can touch, and what they can observe. The last part is the one people forget to write down. For an adversary who holds no weights, has no write path into a training corpus, and is not an enrolled participant in training, the *only* observation channel is the endpoint's reply. That makes the response schema a security artefact: the list of fields you return is, quite literally, the definition of what your weakest-vantage adversary can see. This is why the phrase 'black-box attack' is under-specified on its own. In the access sense used here, black-box means the attacker does not have weights, architecture and gradients. But two black-box attackers can be an order of magnitude apart in capability purely because one endpoint returns a number and the other returns a word. ## The ladder of reply classes Take a hosted content-moderation service for a social product: a third-party site posts user text, the service returns a policy decision, and calls are metered. Its response contract could sit anywhere on this ladder, from most to least generous: - a confidence for **every** policy category; - the top-k categories with confidences; - only the category that fired, with a confidence; - only the category that fired; - a single allow/block bit. Each step down removes a continuous, comparable quantity and replaces it with a discrete symbol. Nothing else about the deployment changes. ## What the generous rungs actually hand over Three distinct things, and it helps to keep them separate: **A direction.** A continuous number moves as the caller edits the input. That lets an attacker tell whether an edit helped *before* the verdict flips, which turns a blind search into a guided one. **A magnitude.** How confident the model is on a given example is correlated with how well it fits that example, which is the raw material of privacy signals such as membership tests. **Cross-category standing.** Scores for the categories that did *not* fire say where this input sits relative to several boundaries at once, not just the one it crossed. Dropping non-fired categories quietly removes most of that. ## The part candidates get backwards The reply class bounds the **cost** of the attacks, not the **list** of them. Families that consume nothing but the returned symbol exist and are well established: decision-based search, which works from inputs whose verdict the model already gives, and label-only membership tests, which read how easily an input's verdict can be made to change. Coarsening the reply pushes an adversary out of the cheap families and into these expensive ones. It moves the price; it does not close the door. 'We only return the class, so there is nothing to leak' is the single most common wrong answer in this area. ## What the schema does not cover Even a one-bit reply is not the whole observation channel. Response latency, error text, whether the service says which policy fired, whether ties are broken deterministically, and how the endpoint behaves at its limits are all observable and none of them are in the JSON body you trimmed. If you coarsen the reply and then claim the observation channel is closed, you have made a claim about one field list, not about the system. ## How to use this in an interview Scope before you attack. When asked 'how would you attack this model', the first move that separates a strong candidate from a weak one is asking what the endpoint returns, because that answer determines whether the engagement is a few thousand queries or a few million. And when reading somebody else's result, treat a robustness, extraction or privacy number quoted without its reply class the way you would treat a latency number quoted without a percentile: not wrong, just not yet a claim.