Transferring an Attack

## What the word "transfer" is actually claiming **Transfer** is the claim that an artefact — here a fixed string of characters produced by an optimisation run — still produces the behaviour it was tuned for on a *different* model, with no further optimisation. Both halves of that sentence are load-bearing: - **"Unchanged"** matters because if you adjust the string against the new target you are running a fresh search, not observing transfer. - **"A model you did not optimise against"** matters because success on the model you searched on is not a result at all; it is the search terminating. ## Phase one: the search, on a model you hold A **gradient-based search** needs weights because its objective is defined on the model's own output distribution — it scores candidate token substitutions by how far each moves the model toward a target continuation and away from a refusal. Every evaluation is a local forward and backward pass. The bill is **GPU hours**: in practice hours of a large-memory accelerator per target model per run, sometimes most of a day for a stubborn objective, plus the engineer time to define the objective, choose the surrogate and babysit the job. What it does *not* cost is API calls. Nothing leaves your network, no vendor's monitoring sees it, no rate limit applies, and you can run it before the engagement window even opens. ## Phase two: the transfer attempt, at a model you do not hold The string is now frozen and you are an ordinary customer sending prompts. Each attempt is one API call — cents, seconds. The entire feedback channel is the reply text plus whatever metadata the response carries. No loss, no logits, no attention, no gradient of any kind. That **asymmetry** is the trap the whole question turns on: the expensive half of the workflow yields something you can iterate on, and the cheap half yields roughly one bit per attempt. Because it is cheap, people run it once and write down a result. ## Why a failure at the hosted endpoint proves so little A hosted endpoint is a **pipeline**, not a bare model. At minimum it is a model plus a system prompt you never see; commonly there is also an input classifier that can reject a request before generation happens and an output classifier that can suppress a completion after it. At least five distinct causes produce the identical blank refusal in front of you: - the model behind the product name is not the build you guessed; - the hidden system prompt reframed the conversation; - an input guard rejected the unusual surface form of an optimised string; - an output guard suppressed the completion; - or the endpoint simply sampled unluckily this time. One reply cannot separate them, and no amount of staring at it will. ## Where the number misleads Two failure modes dominate reports. - **The first is the missing denominator:** one attempt is not a rate, and hosted endpoints sample, so the same frozen string can succeed and fail on consecutive calls. Writing "it works" or "it doesn't work" from a single call is the most common defect in this workflow. - **The second is promotion** — turning "did not transfer" into "the target is not vulnerable to this attack class". That sentence is a claim about a model; your evidence concerns one frozen artefact, against one deployment, on one day, judged by one criterion. The local failure and the hosted failure look alike on the terminal and are epistemically nothing alike: the local one is a search result with a loss attached that tells you where to go next, and the hosted one is a single uninstrumented observation. ## What you would check before believing either outcome - Fire the string enough times to have a **denominator** worth quoting, and record hits and attempts as raw counts rather than a percentage. - Record the **decoding settings** you sent, and note explicitly which ones you could not control. - Capture any **build or version identifier** the response carries. - Store **reply text verbatim** for successes *and* failures: a uniform canned policy sentence, returned faster than a normal answer with nothing streamed, is the signature of a guard in front of the model, while a varied, in-voice refusal is the model itself — and that distinction is unrecoverable later from a tally. - Send the same underlying request in ordinary language as a **control** so you know whether anything at all about the topic gets through. Then write the finding at the resolution your evidence supports, and no further.