Operating a Campaign

Method selection on an engagement is an **access audit first and an algorithm choice second**. Before you name a search, write down exactly what the endpoint hands back, because that one fact eliminates a whole class of methods and prices the two that remain. ### The three things a deployment can give you - **Parameters** — the weight tensors, the matching tokenizer, and the model config, on a disk you control. With these you can run a *forward pass* (tokens in, a next-token probability distribution out) and a *backward pass* (a loss differentiated with respect to the inputs or the weights). - **Scores** — numbers that describe the reply without being the reply: per-token log-probabilities, a top-k distribution over next tokens, or a moderation classifier's per-category confidence. - **Text** — the completion string, and perhaps a stop reason. Nothing else. A metered chat API almost always gives you the third, sometimes the second, and never the first. ### What each search class actually reads **Gradient-guided discrete token search** — the published GCG family — defines a loss as the negative log-likelihood of a chosen target continuation given the prompt plus an appended suffix, then backpropagates that loss into the embedding positions of the suffix. The gradient does not pick a token; it *ranks* candidate substitutions at each position, and the algorithm then evaluates a batch of the top-ranked candidates with real forward passes and keeps whichever actually lowered the loss. Both halves of that loop require the parameters in memory you control. An HTTPS endpoint is a function you may *evaluate*; it is not a function you may *differentiate*. There is no backward pass through a POST. Text-only access rules this class out before a line of code is written. **Attacker-model-in-the-loop search** — the PAIR and tree-of-attacks families — reads reply text only. One model proposes a candidate prompt, the system under test answers, a judge model rates the answer, and the attacker rewrites with that feedback in its context. Every signal it consumes is a string, so a text-only key is sufficient. **Template and mutation pools** need no model on your side at all. A curated seed corpus is expanded by mechanical transforms — paraphrase, encoding, role framing, language switch — and each variant's reply is graded by a keyword rule or a small classifier. ### What each costs | class | the meter | what you actually spend | |---|---|---| | gradient search | GPU-hours and VRAM | nothing per attempt; hours to days of a machine you own, per suffix | | attacker loop | three billed endpoints | attacker, target and judge calls every iteration, with token cost rising as the transcript grows | | template pool | one billed call per variant | near-zero engineering; the maintained pool is the asset | Note that the binding limit moves from GPU memory to **the request meter and the rate limit**, which are two different limits. A campaign can be fully funded and still impossible, because a per-minute ceiling turns a depth-heavy loop into a fortnight of wall-clock. ### Where the reading goes wrong - **"Black box" and "text-only" are not the same claim.** If the endpoint returns log-probabilities or a moderation confidence, you hold a continuous objective, and blind accept/reject sampling can be replaced by score-guided hill-climbing that typically reaches a hit in far fewer requests. Calling the target "black box" and stopping there leaves that on the table. - **A same-named local checkpoint is not the target.** The deployed product is a stack: a hidden system prompt, possibly retrieval and tools, an input classifier, an output filter, and a served checkpoint that may be tuned or quantised differently. None of that is in a download. - **A zero-finding pool run reads as robustness and is not.** The denominator of a template pool is the pool. "No hits across 400 templates" means nobody seeded the thing that works, not that nothing works. - **Billed is not tested.** A request rejected by an input filter before generation still costs money and still fails to tell you anything about the model, so a per-request cost figure overstates the coverage it bought. ### What to check before committing Does the response body carry log-probabilities or a moderation score, or only text? What are the per-minute and per-day request ceilings, and does the planned depth fit inside them? Is the object under test the model or the deployed system, and does the scope statement say which? Is sending client-derived prompts to a third-party attacker or judge model contractually permitted at all — because that single clause can eliminate the loop class and leave you with pools.