How heavily should trajectory metrics gate an agent release without freezing it onto one path?

## The failure this question is about Trajectory metrics are so much more informative than a pass/fail bit that the natural instinct is to gate on them. That instinct, followed all the way, produces an agent optimized to reproduce the trajectories your team wrote down eighteen months ago. Every improvement that arrives as a *shorter or different* path — a stronger model that skips a lookup because it can infer the value, a new bulk tool that replaces three calls with one — is scored as a regression. The eval stops measuring quality and starts measuring similarity to the past. ## A three-tier gate **Tier 1, blocking.** Only properties you would defend as invariants of correct behaviour: - outcome correctness on the task suite, against a baseline threshold; - forbidden-call violations — an unauthorized side-effecting tool, a call outside the tenant's scope; - required-call violations — answering a factual question with no retrieval in the trace; - hard ceilings on steps, wall-clock and cost per episode, set well above normal so they catch runaway behaviour rather than ordinary variation. These share a property: each is a rule about what must or must not happen, independent of how the agent chooses to work. **Tier 2, budgeted.** Efficiency signals — median and p95 step count, redundant-action rate, tokens per successful task. Gate these against a rolling baseline with an explicit tolerance (say, a 25% degradation in median steps) rather than an absolute number, so the check tracks the system instead of a snapshot of it. Budget breaches should require a human decision, not an automatic block: sometimes the extra steps buy a real quality gain and the right move is to move the baseline. **Tier 3, advisory.** Path-similarity scores against reference trajectories and judge rubric criteria. Publish them per run, alert on trends across runs, and review them when something else is already suspicious. Never let a single-commit movement in these numbers block a merge, because their run-to-run noise is comparable to the effects you would be reacting to. ## The maintenance economics nobody budgets for Reference trajectories are code, and they rot faster than code. Every tool rename, schema change, added capability or model upgrade can invalidate a batch of them. A suite of a few hundred references represents a standing maintenance obligation, and when it exceeds what the team will actually pay, the observable outcome is not that references get updated — it is that failures get waved through, which is worse than not having the check. Before committing to reference-based gating at scale, cost the upkeep honestly and prefer encodings that survive change: constraints and budgets referencing *capabilities* age far better than full call-by-call transcripts. A cheaper middle path is targeted references. Author full reference trajectories for the handful of flows where the path genuinely is the product — regulated procedures, destructive operations, anything with a compliance story — and use constraints plus outcome checks everywhere else. That concentrates the maintenance where it earns its keep. ## Tasks with no single right path Many tasks admit several equally good solutions; research and open-ended investigation tasks admit dozens. For those, a reference is not merely expensive, it is conceptually wrong — there is nothing to be the reference *of*. The honest instruments there are outcome checks on the artifact produced, invariant constraints, budgets, and a rubric applied by a judge. Trying to force a canonical path onto such tasks is how teams end up with evals that are simultaneously strict and uninformative. ## Deciding the weights A workable heuristic: a trajectory metric earns blocking status only if you can state the user-visible harm of violating it in one sentence. "It issued a refund it was not asked for" — block. "It exceeded the cost ceiling by 8x" — block. "It took a different but valid route to the same answer" — that is not harm, and gating on it buys nothing while costing you every future improvement that looks unfamiliar. Also plan for the gate's own evolution. When a model upgrade legitimately changes the shape of good trajectories, the correct response is to re-baseline deliberately — rescore a window of known-good runs, move the budgets, retire references the new capability makes obsolete — and to record that you did, so later comparisons are not silently made across an inconsistent baseline. A gate nobody is allowed to move eventually gets bypassed instead.