Automated Jailbreak Generation

Hand-label a random sample of the sixty labelled hits to get precision. For the other direction you need the stored transcripts of everything the scorer rejected: sample those, weighted toward borderline confidence, and hand-label. If only hits were kept, say recall is unmeasured rather than guessing.

Only when you treat the result as a candidate, not a finding: a suffix optimised against local weights is tuned to that copy's tokenizer and parameters. Reserve enough requests against the real product to test transfer, and expect most candidates to fail there, especially through an unseen system prompt and output filter.

Stop paying for repeats. First check the flat stretch is a real plateau, not a stalled judge or a collapsed search. If it is real, spend the remaining allowance on a different seed set or objective rather than more of the same, and record the queries spent and the last new template so the stop is evidence.

It measured that one search stopped finding new templates, not that the target has none left. Saturation is relative to the seeds, objective, scoring step and mutation operators that run used; change any of them and the reachable region changes. Word it as this search exhausted, with those settings named, never as an absence of weaknesses.

The corpus survivorship-collapses. Dropping seeds that stop landing leaves only the family the current model version is weakest to, so the hit rate measures your corpus, not the model. Keep a frozen regression set that includes the dead seeds, quota seeds by family, and reserve part of every campaign for families the corpus has never held.

Only one precondition held: you had weights you could differentiate. You optimised a different numerical build, on an input that omits the production preamble, for a path that has no classifier in front. Before more GPU hours, load the quantised build, include the real system prompt as fixed context, and search only the attacker-controlled region.

Set it from the GPU hours you can spend divided across the behaviours you must cover, not from a number in a paper. Watch the loss curve and run the real success check periodically: stop the moment a verified hit lands, and restart from a fresh initial suffix when the loss has plateaued rather than buying more steps.

Run shallow across everything first, then spend the remainder deep on the behaviours that showed partial movement. Wide answers which behaviours are reachable at all and gives an honest coverage denominator; deep answers how hard a specific one is. Decide by which claim the report has to support, and write the split down beforehand.

You can write that no attempt succeeded within the turn cap and restart count used, against this seed set, judged by this scoring model, on this target build and date. You cannot write that the target is jailbreak-resistant or that no attack exists. A capped search reports a budget, not a property.

Standardise the definition of a distinct template, the scoring method, and the requirement that every stop cite a yield curve and the queries spent. Leave the threshold and the reseed decision to the operator, since targets differ. Get it backwards and teams tune the clustering until every campaign saturates exactly on schedule.

Treat mutation fuzzing as cheap regression monitoring on a fixed cadence and cap it, because its marginal value decays fast once seeded families are mapped. Steer on new distinct families per thousand queries, not on hits. Spend the freed budget on discovery that authors families the corpus never held, and report untested families explicitly.

Require a written success criterion: what behaviour counts as a hit, what scores it during the search, and what independently confirms it afterwards. Require a cheap pilot showing the signal separates known hits from known-benign prompts. Accept confirmed, reproducible, verified prompts, not a peak fitness curve.

Decide from the target mix: the capability only applies where you hold weights, which here is one or two systems. Beyond hardware you take on weight custody and deletion duties, licence review, per-target GPU hours that never amortise, re-runs after every checkpoint update, and staff time to keep the rig matching each serving stack.

It supports layered defence: input anomaly checking, output-side review, and a regression case kept for future checkpoints. It shows refusal training does not hold off-distribution. It does not support severity claims about a live surface, statements about systems you hold no weights for, or any headline that the model is broadly unsafe.

Decide from your own history, not from the literature: track what fraction of locally optimised strings ever fired at a hosted endpoint and how long they kept working. If that half-life is shorter than your reporting cycle, transfer buys exhibits that are stale on delivery. Keep it where it produces cheap candidates; cut it when the endpoints are guard-dominated.

A scorer that shares the target's blind spots misses exactly the outputs the target should have refused, so the most important successes go unreported. Prefer an independent scorer, add deterministic detectors, measure the scorer against a fixed human-labelled set, and require human adjudication before any finding is signed off.

Standardise on what most engagements can actually run: template and mutation pools for cheap breadth, and an attacker-model loop for depth, both needing only text access. Keep white-box search as an occasional capability. You lose the ability to attack open-weights deployments at their strongest and to produce transferable candidates cheaply.