How often should a golden eval set be refreshed, and what should trigger a refresh?

## Four ways a golden set goes stale **The world changed.** The expected answers encode facts and rules that live outside your system: a formulary update, a policy revision, a pricing change, a regulation. When those move, the labels are wrong while the items still look perfectly fine. This is the most dangerous form of staleness because it inverts the metric — the system now loses points for being right. **The product changed.** New capabilities, new tools, new supported formats. The set exercises the old surface and says nothing about the new one, so coverage silently drops even though the score holds steady. **The traffic changed.** The set was stratified against a distribution that has since shifted — a new customer segment, a new channel, a seasonal pattern. Slice weights that made the headline representative no longer do. **The set was overused.** Items iterated against for months stop being a measurement. Every prompt tweak that was accepted or rejected on this set has fitted the system to it, and the residual headroom on the set no longer predicts headroom in production. ## Cadence plus triggers A purely scheduled refresh is too slow for rule changes and too aggressive for stable domains. The workable policy is both: - **Scheduled review** — quarterly is a common default. The review is not automatically a rewrite; it is a checkpoint that asks whether the labels are still correct, whether the slice list covers current failure modes, and whether the traffic mix still matches the strata. - **Event triggers** that force a refresh regardless of the calendar: an external rule or policy change that touches the labeled decisions; a shipped capability the set does not exercise; a measurable shift in the request mix; a cluster of production incidents in situations absent from the set; a rubric change from an adjudication round. ## Refresh means a new version, not an edit Silently editing items is the most common process failure. Scores from before and after the edit are then quietly incomparable, and nobody can tell whether a step change came from the system or from the set. The discipline: 1. Cut a new set version with a version identifier, a changelog of items added, removed and re-labeled, and the rubric version it was labeled under. 2. **Re-score the previous system version on the new set** to re-establish a baseline. Without that, the first comparison after a refresh measures the set change, not the system change. 3. Keep the old version retrievable, so historical claims remain auditable. A useful supplement is a small, deliberately frozen archival slice — items chosen because their correct answer is timeless — kept unchanged for years so long-run trend has at least one stable anchor. ## Distinguishing a moved yardstick from a real regression When a score drops, the first question is which of the two moved. Cheap discriminators: did the drop concentrate in exactly the categories the external change touched; do the failing items' *expected* answers still look right to a reviewer; does the previous system version, re-run today, also score lower on the same items. If the old version regresses too, the set changed, not the system. Building that check into the refresh routine prevents the classic incident where a team spends a week debugging a model for a formulary update. ## Churn budget and the maintenance economics Refresh is not free — it is re-labeling, re-adjudicating, and re-running baselines. A useful governance device is a churn budget: an expected fraction of the set that may change per cycle, say 10 to 20 percent, with anything larger requiring an explicit decision. Too little churn and the set ossifies; too much and no metric survives long enough to show a trend. The deeper point is ownership. An eval set with no named owner and no funded maintenance decays exactly like documentation, except its decay is invisible until it produces a wrong decision. Whoever owns the product decision the set informs should own the set, and the refresh cost should be a line item in the same budget as the feature work it gates. ## Interaction with over-fitting Because months of iteration fit the system to the set, refresh also serves a hygiene role: rotating in fresh items restores headroom and re-exposes weaknesses the team has been implicitly tuning around. Keeping a portion of each refresh untouched by any iteration — added, labeled and reserved — gives you a clean surface for the decisions that actually matter, at the cost of measuring less often on it.