Two CI runners build the same commit with different Go toolchains — how do you diagnose that drift?

## Why the drift is possible at all Because the go command may hand off to a different toolchain, "which Go built this" has two answers: the one installed on the machine, and the one that ran. Under `GOTOOLCHAIN=auto` a runner with an older installed Go will switch up to satisfy the repository, while a runner already at or above that version will not switch and will use its own — which may be a *different, newer* release than the one the first runner downloaded. Same commit, two compilers, no error anywhere. ## The inventory to collect Run the same three commands on every machine that builds, developer laptops included, and store the output next to the machine name: - `go version` — the toolchain that responded to the command. - `go env GOTOOLCHAIN` — the effective setting. This is the important one, and it must be `go env`, not the shell variable, because the value can come from three places. - `go env GOMODCACHE` — where downloaded toolchains land, which tells you whether a runner has been switching and caching releases at all. The expected shape of the result is boring uniformity. Any row that differs is your answer. ## Where the setting comes from Precedence: the process environment, then the go environment file written by `go env -w`, then the default compiled into the distribution. Three practical consequences: 1. A job or a shell profile that exports `GOTOOLCHAIN` beats everything, and is invisible unless you look at the job's environment. 2. An image built by someone who once ran `go env -w` carries that choice forever, in a file no Dockerfile mentions. 3. Official Go releases compile in `auto`, but a Go obtained through some other packaging may not. Two images that both report a plausible `go version` can still disagree on policy. ## Evidence from the artefact The machine can be reimaged; the binary cannot lie. `go version -m ./yourbinary` prints the build information embedded in a Go binary, including the Go version that produced it along with module and build settings. Comparing that across the two runners' artefacts confirms the drift independently of whatever the runners report today, and it is the check to bake into a release pipeline: assert that the shipped artefact was built by the release you intended. ## The repository side If the machines agree, look at the checkout. The `go` line sets the floor that triggers a switch; the `toolchain` line names the destination; and in a workspace, `go.work`'s own lines take precedence over the individual modules'. A `go.work` file present on one runner and not another — checked in on one branch, generated by a setup script on the other — will produce exactly this symptom. ## The failing-runner variant The same root cause has a louder form: one machine builds and another refuses. That is `GOTOOLCHAIN=local` (or a machine that cannot reach the module mirror to fetch the toolchain) meeting a repository whose `go` line has moved ahead. The error names the required version, the running version and the setting, so it diagnoses itself — the interesting part is that it is the *same* configuration drift, surfacing loudly on one machine and silently on the other. Preferring the loud form is a legitimate design choice. ## Fixing it for good The fix is to stop letting the answer depend on what happened to be preinstalled: - Set `GOTOOLCHAIN` explicitly in the build image — an exact version name for reproducibility, or `local` if the image is supposed to be the sole source of the compiler. - Make the image's installed Go and that setting one change: bumping the compiler should be a single reviewed diff. - Record the toolchain in the repository with a `toolchain` line if you want repositories, rather than images, to lead — but know that a pinned image overrides it. - Verify, don't assume: have the pipeline print `go version` and `go env GOTOOLCHAIN` into the build log, and check `go version -m` on the artefact before release. A one-line log statement turns this class of incident into a five-second diff. ## What good looks like in the interview Say that you would gather evidence from machines *and* artefacts, name the three-level precedence of the setting, and end with a policy change rather than a one-off fix. Blaming "the runner image" without saying how you would prove it is the weak version of this answer.