The Cost of Gradients

## The mechanism, one iteration at a time A **gradient-guided prompt search** — the family that includes published methods such as GCG — treats the prompt as the variable being optimised and holds everything else fixed. One iteration does three things. - (1) *Forward pass.* The candidate prompt is tokenised, mapped to embedding vectors, and run through the network to produce a scalar loss. The loss is usually the negative log-likelihood the model assigns to a target continuation the attacker picked in advance — a compliant-looking opening phrase — so "lower loss" means "the model is more inclined to begin the way I want instead of refusing". - (2) *Backward pass.* Automatic differentiation propagates that loss back through every layer to the input embedding matrix, yielding a gradient vector at each attacker-controlled token position: a local, linear estimate of which direction in embedding space lowers the loss. - (3) *Candidate ranking and re-scoring.* Because the search space is a discrete vocabulary rather than a continuous vector, the gradient cannot be applied directly; it is used only to shortlist plausible token substitutions, and each shortlisted candidate is then re-scored with a further forward pass. Step 2 is the step that requires the **parameters**. It is not optional dressing on the method — it is the method. ## What a hosted endpoint actually returns A text-in/text-out chat API gives you - generated tokens, - sampling controls such as temperature and top-p, - sometimes a top-k log-probability field per generated token, - sometimes moderation labels, - and sometimes an embeddings endpoint that serves a *different* model entirely. Not one of those is a derivative of the serving model's loss with respect to your input embeddings. There is no request shape that turns a completion into a backward pass, because the vendor never computes one for your request and would have to expose its parameter graph to do so. ## Why "just estimate the gradient from queries" is not a workaround, in numbers **Derivative-free estimation** over a discrete vocabulary means comparing alternatives by measuring them. - A modern tokenizer has roughly 32,000 to 256,000 entries; - a control span is typically ten to thirty positions; - a single well-informed step compares thousands to tens of thousands of alternatives; - and this class of search converges, when it converges, over hundreds to low thousands of steps. Multiply and you are in the millions-to-tens-of-millions of endpoint calls **for one behaviour against one model**. At even a fraction of a cent per call that is a five- to six-figure bill, and at a typical commercial rate limit it is months of wall clock — before the vendor's abuse telemetry notices tens of thousands of near-identical adversarial prompts from one key and suspends it. That cost gap is exactly why the **query-driven families** exist: an iterative-rewrite loop, where a second model rephrases a failed prompt and retries, converges or gives up in tens of queries, not millions. ## Where the number misleads Two readings go wrong, and both matter more than the mechanism. - First, the *silent null*: a report that says "automated jailbreak search produced no successful attack" against a query-only target is describing the team's access, not the model's robustness. Nothing ran. A reader — often an executive or an auditor — converts that sentence into "the model withstood optimisation-based attack", which is a claim no one made and no evidence supports. - Second, the *quiet substitution*: a team runs the search against an open-weights model of similar architecture and reports the hits it lands on the real target as though they were results of a search against it. That is a **transfer measurement**, with a much lower and much more fragile success rate that decays with every checkpoint update on the target side, and it must be labelled as such. ## What you check before committing - Read the access grant, not the statement of work's adjectives: does the engagement deliver a checkpoint file, or only an API key? - If a checkpoint, is it the build that serves traffic, and are you licensed and contractually permitted to hold it? - If only a key, write the method plan as query-driven from the start and cap the query budget against the vendor's rate limits and the client's spend. - Confirm in writing that automated adversarial probing of that endpoint is authorised, since a third party's terms of service usually forbid it by default. - Then, in the report, name the families you could not run and why, so the clean result is read as the bounded thing it is.