How do you prove a service still needs its GODEBUG opt-out after a Go upgrade?

## Why this needs measuring at all A GODEBUG opt-out is invisible by construction. It restores the behaviour your code already expected, so nothing in the service misbehaves, no log line appears, and no test fails. The result is that opt-outs outlive their reason by years: the person who set it has moved on, nobody can name the code path that needed it, and the team's honest answer to "can we remove this?" is "nobody knows". The Go runtime anticipated exactly this and instrumented it. ## The counter For each compatibility setting there is a metric named: ``` /godebug/non-default-behavior/panicnil:events /godebug/non-default-behavior/asynctimerchan:events ``` The important word is *behavior*. It does not count how many times the setting was read or whether it is set — it counts occasions on which the program actually behaved the old way **because** the setting was non-default. If your service sets `panicnil=1` and never panics with a nil value, that counter stays at zero and the setting is doing nothing for you. That makes it a decision instrument rather than a curiosity: - **Non-zero, steady:** you genuinely depend on the old behaviour. The rate tells you how hot the path is, and the counter's existence tells you exactly which behaviour to go and fix. - **Non-zero, rare and spiky:** the dependence lives on an unusual path — an error branch, a retry, a scheduled job. This is the case that would have made a code-review-only removal an outage. - **Zero over a full cycle:** the strongest evidence you can get that removing the setting is safe. ## What a full cycle means This is where teams get it wrong. A day of steady traffic is not a cycle. Include the month-end or quarter-end batch, the nightly reconciliation, the deploy path, the failover drill, and the error paths that only run when a dependency is down. A leaked goroutine or a timer edge case may only be reached when something else is already failing, and "we watched it for 24 hours and it was zero" is precisely how that gets missed. Aim for at least one full business cycle, and prefer instrumenting once and leaving the counter on a dashboard for months over running a one-off check. ## Two supporting techniques **A/B the fleet.** Because the environment variable overrides whatever the module declares, you can start a fraction of instances with the setting flipped to the new behaviour while the rest stay on the old one, and compare error rates and the affected behaviour directly. It is a cheap, reversible experiment: no rebuild, and rolling back is a restart. Keep the two halves comparable — same traffic mix, same version otherwise — or the comparison proves nothing. **Pin the exit date to the toolchain, not to a sprint.** Compatibility settings are kept for a documented window and then deleted along with the old code path. When that happens the setting stops having any effect, and it does so quietly rather than by failing the build or the boot. `asynctimerchan`, the opt-out for the Go 1.23 timer-channel change, was removed in Go 1.27; a fleet still leaning on it simply got the new behaviour the day it upgraded. So the removal ticket should carry the release that deletes the setting, and the upgrade checklist should ask whether any setting in play is on its last release. ## Putting it together as a routine 1. Every compatibility setting you add is recorded in the module and paired with its counter on a dashboard. 2. The counter is reviewed at each toolchain upgrade, not on demand. 3. Zero across a full cycle plus a clean A/B is the standing bar for deleting the setting; a test that pins the new behaviour goes in with the deletion so nothing quietly reintroduces the dependence. 4. Non-zero means the fix is scheduled with a named owner, because the deadline is set upstream and will arrive whether or not the work does. The habit worth carrying out of this is the general one: an opt-out with no measurement attached is not a decision, it is a deferral with no end date.