When is setting a GODEBUG opt-out in production the right call, and who owns removing it?

## The decision, stated honestly A Go upgrade lands. Under real traffic, something behaves differently from before and a service starts erroring. Two levers are within reach: undo the upgrade, or set the documented setting that restores the old behaviour for that one change. This is a judgment call, and it is owned by whoever runs the service — but it is a call the platform team, which is trying to move a fleet forward, can and should push back on. ## Why the narrow lever usually wins A rollback is broad. The upgrade carried compiler fixes, standard-library fixes and, very often, security fixes; reverting it takes all of them back, and if application changes shipped in the same artefact you may not have a clean version to return to. It also stops the fleet-wide upgrade dead, which matters when half the fleet is already on the new release: now you are running two behaviours at once, and every subsequent diagnosis has to account for which half a request landed on. A named compatibility setting is narrow, documented and reversible. It restores one behaviour whose scope is written down, leaves everything else in the upgrade in place, and is undone by a restart. It also *un-splits* a half-upgraded fleet: set it on both halves and the observable behaviour matches again, which is what makes the rollout safe to continue or to reverse deliberately rather than in a panic. The cost is that you have chosen to keep depending on behaviour Go has decided to retire, and the retirement date is not yours to move. ## Making the debt visible The difference between a defensible opt-out and one that becomes an outage in two years is entirely in what you do in the hour after the incident. **Put it in the repository.** An environment variable in a deployment manifest is invisible to tests, differs between laptop, CI and production, and never appears in a code review. A declaration in the module travels with the code, applies to the test binary, and shows up in the diff with a commit message that says why. Keep the ability to override at run time — that is what saved you tonight — but the durable record belongs in the module. **Name one setting, not a blanket default.** The form that moves every compatibility default at once is tempting during an incident because you do not yet know which behaviour bit you. As a lasting state it is much worse: it freezes many behaviours, so nobody can tell what you actually depend on, and eventually unfreezing changes all of them simultaneously. If you use it, use it for hours, not quarters, and narrow it as soon as the counters tell you which setting matters. **Instrument it the same day.** The runtime's per-setting counter of occasions where behaviour actually diverged is the only thing that later turns "nobody knows" into a decision. A setting added without its counter on a dashboard is a setting nobody will dare remove. **Give it an owner and an upstream deadline.** Compatibility settings are removed after a documented window. When that release lands, the setting stops having any effect and the behaviour changes anyway — quietly, with no build failure. So the removal ticket carries a release, not a sprint, and a person, not a team alias. ## Where the organisation gets a vote A single service's opt-out is local. A dozen services each holding a different one is a fleet that cannot be upgraded on a predictable cadence, and that is a platform-level problem. Reasonable standing policy: opt-outs are allowed and expected during upgrades; each is declared in the module, instrumented and reviewed at every toolchain upgrade; blanket defaults require an explicit, dated exception; and any setting entering the last release before its removal escalates to a scheduled fix with engineering time allocated. The platform team's authority here is not to forbid the flag — forbidding it just produces rollbacks and stalled upgrades — but to insist the debt is recorded and time-boxed. ## The tradeoff nobody should skip Setting the flag buys time and costs optionality: you are now on a clock set by someone else's release schedule, and the work to get off it will not get easier. Rolling back buys certainty and costs everything else in the upgrade, including fixes you may care about more than you realise at 3am. State that tradeoff out loud when you make the call, and write down which side you chose and why — the next person to touch the setting will be reading exactly that note.