You own automated jailbreak campaigns across a dozen deployed targets, run by several operators, and you must publish one stopping rule based on distinct-template yield. What do you standardise, what do you deliberately leave to the operator, and what goes wrong if you get that split backwards?

## The number that gates work becomes a target This is a measurement-governance problem wearing a red-team hat. The moment distinct-template yield decides when a campaign stops, it stops being a passive observation and becomes something people steer towards — rarely through dishonesty, usually through a hundred individually defensible small choices about what counts as the same template. Whatever the operator can adjust to satisfy the gate will drift until it satisfies the gate. So the design question is not "what is the right threshold" but "which knobs may an operator hold". ## Definitional versus economic settings | kind | examples | why it falls this way | |---|---|---| | Definitional — standardise | the clustering rule that defines a distinct template; the scoring method that accepts a success; the curve's units; the freeze-before-run requirement; the record every stop must carry; the mandated wording of the conclusion | these fix what the number *means*. If they float, two campaigns' numbers are not the same quantity, and no cross-target comparison, trend line or funding decision built on them is valid | | Economic — devolve | the no-new-template window; the query allowance; seed corpora; objectives; whether a flat curve means stop or reseed | these encode how much a particular target is worth spending on, and that genuinely differs. A high-traffic consumer surface and an internal tool with ten users do not deserve the same window | Getting the split backwards is the classic failure: mandate one global threshold while letting each team define what a distinct template is, and you have handed every operator a dial that makes their campaign finish on schedule. The gate is fixed, the definition is adjustable, and the definition is what moves. ## What to write into the standard - **One clustering rule, versioned**, with the version stamped on every campaign record, so a later revision of the rule does not silently rewrite the fleet's history. - **One scoring method**, with a mandated periodic hand-audit of a sample of accepted successes. - **Curve units** — distinct templates against queries spent — and the requirement that rule and threshold are frozen before a query is spent. - **A record template**: queries spent and the billed-call multiplier behind them, the curve, clustering-rule version, seed corpus, objective, last new template and when it arrived, diagnosis, stop reason. - **Mandatory wording**: an exhausted search, with its configuration named. Never a clean target. - **More than one configuration per target per cycle**, because a single plateau is a single sample. ## What the standard itself costs State it, because an unfunded standard is a standard that gets skipped. The hand-audit is engineer time: a sample of accepted transcripts per campaign per cycle, read by someone competent to say whether the behaviour is actually present — the largest recurring line item, and the one teams drop first. Rotating a second scoring method through a fraction of campaigns adds judge calls on re-scored successes, a small single-digit percentage of a campaign's spend, since only accepted responses are re-scored rather than every attempt. A deliberately off-standard control configuration costs a whole extra allowance on the targets that get one. Against these, the cost of *not* paying: yields that are not comparable, which means every fleet-level chart is decoration. ## Where the fleet number misleads **A tight band.** If campaigns across the fleet stop within a narrow range of queries spent, that is a smell, not reassurance. Real discovery curves are ragged and targets differ; a fleet that saturates on schedule is reporting its schedule. **Trend lines that track rule versions.** A quarter-on-quarter yield chart will happily encode a clustering-rule revision as a change in the targets. This is exactly what version stamping is for, and the chart should be broken at every version boundary rather than smoothed across it. **Correlated blind spots.** The honest cost of centralising definitions is that every campaign inherits the same clustering rule and the same scoring step, so a weakness in either is invisible everywhere at once. Pay it anyway — incomparable numbers are worth nothing — and spend against it deliberately: hand-audits, a rotated second scoring method, and one configuration kept off-standard as a control. **External reports.** When an outside submission names a mechanism no in-house campaign ever produced, the evidence says the shared reachable region is too narrow. The response is more configuration diversity — disjoint seed corpora, different objectives — not tighter thresholds or longer runs, which only deepen the region you already had.