The team receiving your guardrail bypass findings also owns the guard's block threshold and can change it the day after you deliver. How do you make the figure you hand over still mean something a month later?

## The frame: this is governance, not statistics The tester measures; the measured party owns the dial. Nothing prevents the dial from moving the day after delivery, and nothing prevents the old percentage from being quoted for a year afterwards. So the deliverable has to be built to survive its own inputs changing. The failure is rarely dishonesty — it is a true number, measured carefully, that quietly stops describing the system while continuing to circulate. ## Four mechanisms, roughly in order of leverage **1. Ship data, not a verdict.** The durable output of the engagement is the run: per-attempt guard scores, the corpus, the verbatim configuration snapshot, and the success judgements. From those, bypass at *any* threshold is a computation that takes seconds. From a slide with "4%" on it, nothing is a computation and the only way to answer "what about at the new setting?" is to pay for the whole run again. Shipping the data also makes the finding auditable: a reader who disputes your judge can re-derive with theirs instead of arguing about yours. **2. Pre-register the operating point.** Agree, in writing, before results are inspected, which threshold the headline number is quoted at — normally the one in production on the day of the run — and why. Choosing it afterwards is selection on the outcome, and it is equally available to a tester who wants a frightening number and an owner who wants a clean one. Pre-registration is what makes the headline a measurement rather than a choice. **3. Make loosening visibly expensive.** A bypass curve on its own has a monotone shape that argues for tightening forever. The only reason nobody tightens forever is refused legitimate traffic, and that cost must come from a benign hard-negative set measured alongside. Present the two together and moving the dial becomes a decision with a stated price in both directions, which is the honest frame for the owner to decide in — and it removes the incentive to treat your bypass figure as an attack on their configuration. **4. Bind the finding to a configuration version.** Ask that the guard configuration be versioned and that changing it invalidate the associated finding, the same way a code change invalidates a passing test. The organisational win is procedural: a threshold edit generates a re-measurement task instead of an argument about whether the old report still applies. ## What the governance costs Pre-registration and versioning cost meeting time, not money. The re-measurement trigger is the item with a real price, and it should be scoped rather than promised blanket. Price it explicitly: one full rerun is *n* guard calls plus *n* target completions plus judging, and a day or two of triage; if the configuration changes monthly, a "re-measure on any change" policy buys twelve of those a year. The workable version tiers it — a threshold change triggers a sampled regression on a fixed slice, a change of guard, model or enforcing categories triggers the full corpus — and the tiering is agreed at the same time as the trigger, or it will be renegotiated under time pressure the first time it fires. ## Where the number misleads once it leaves your hands - **The summary layer eats the operating point.** Each time the finding is condensed on its way up a reporting chain, the qualifier is the first thing dropped and the percentage is the last. Defend against it by putting the operating point *inside the sentence* — "3% at high strictness with these categories enforcing" — so a copy-paste carries it. A footnote does not survive a slide. - **Trend charts plot config edits as hardening.** Quarter-on-quarter improvement is only improvement if the dial held still. Any comparison across runs must state whether the operating point was the same, and refuse the comparison outright when the guard version changed underneath. - **A frozen client-side config does not freeze a hosted guard.** If the operator can retune the model behind the endpoint, even an unchanged configuration on the client side does not preserve the measurement. The honest statement is that the result has a shelf life; name it and schedule the re-measurement rather than letting the number age silently. - **The curve without its benign counterweight is an argument, not a report.** Delivered alone it reliably produces "so we should tighten it", a recommendation your data does not support. ## What to check after delivery Two concrete checks. First, ask the receiving team to reproduce a single row of your curve from the artefact you handed over. If they cannot run the recomputation, you shipped a PDF and the whole mechanism is decorative. Second, read the executive summary somebody else wrote from your report and see whether the operating point survived the condensation; if it did not, that is a formatting fix you can still make, and it is the difference between a number that means something in a year and one that merely persists.