You own the rule for how third-party safety-benchmark scores may be cited in your organisation's model-vendor reviews. How do you decide how long such a published number stays citable, and what do you require once it has expired?

### Set what the number may decide, and the interval follows The policy question is not "how many months is a published safety score good for" but "what is this number allowed to decide". Fix that first and the expiry rule almost writes itself, because the tolerance for staleness is a function of the stakes, not of the benchmark. **Tier the uses.** | use | what may be cited | why | |---|---|---| | context and shortlisting | any published score, labelled with its measurement date, harness and judge | cheap, low stakes, and honest about what it is | | comparing vendors | only numbers from the same benchmark revision, same judge, comparable harnesses | in practice this means you produce them yourself; two published rows are two different experiments | | gating a launch or renewal | an in-house run against your integration, within the current cycle, with a manifest and transcripts | the published run drove a bare endpoint; yours has a system prompt, tools and retrieval | The third row is the load-bearing one. No third-party number, however recent, clears a launch gate — not because it is untrustworthy, but because it measured a different system. ### Two clocks, and the earlier one wins **The event clock is the real one.** A published number expires the moment something it silently assumed changes: the vendor updates the model or its serving-side filtering; the benchmark ships a revised behaviour set or a new harm judge; your own system prompt, retrieval corpus or tool surface changes materially; or a technique lands publicly that the suite never contained. Any of these moves the true rate without a date passing. Name an owner for each trigger — vendor-comms watcher, benchmark-release watcher, the platform owner for your own integration changes — or none of them will ever fire. **The calendar backstop exists because the event clock depends on disclosure you do not get.** Most vendor-side changes are never announced to you at all, and provider-side filtering usually sits outside the model changelog. So add a flat expiry: a quarter for anything feeding a decision, longer for background context. Shorten it for vendors who will not answer questions about change. ### What the policy costs, and the failure mode of writing one you cannot fund A quarterly full re-run is not free. A 200-behaviour suite at five attempts is roughly 1,000 target completions plus 1,000 judge calls per vendor per cycle — tens of dollars of endpoint spend, which is negligible — and then half a day to a day of analyst time triaging flagged transcripts, which is not. Multiply by the number of vendors under review and by four cycles a year and the human line item is what decides whether the policy is real. A cadence the budget cannot fund quietly becomes fiction, and fiction is worse than a longer honest interval, because people keep citing the stale number while believing the policy protects them. Two levers when the budget binds. **Cut coverage, not cadence**: keep a stable, fixed subset plus a control running on schedule, and reserve full coverage for triggered events and the annual review. A small suite run reliably beats a large one run once. **Automate the manifest and the diff**, so the recurring cost is triage rather than setup. ### Where the numbers mislead, and what the policy must prevent The characteristic failure is **citation laundering**: a number that was true on its measurement date is quoted in a review, then copied into the next quarter's deck with the date dropped, and by the third quarter it has become a standing fact about the vendor. Prevent it structurally — require the measurement date, harness and judge to travel with the number in the template itself, so a bare figure cannot be pasted. The second failure is the **smoothed series**. When the benchmark revises its behaviour set or you change judges, the series breaks. Draw one line through the break and a harness change becomes a claimed safety improvement two quarters later. Mark discontinuities in the tracker. ### What you would check, and what to demand of vendors Before a number enters a review: does it carry a date, a harness description and a judge identity; is its tier recorded; is an owner named for each trigger that would expire it. Of the vendor: the measurement date and harness behind any figure they quote, whether the endpoint has changed since, and whether they will notify you of safety-relevant updates. The answers are often unsatisfying — and "this vendor cannot tell us when the model changes" is itself a finding worth writing down, because it sets a floor on how stale your evidence can silently become.