skip to content

Your published module's v1.3.0 tag was force-moved to a new commit and consumers now fail verification. Why?

level: seniorimportance: should knowfreq 45%

answer

  1. a version means content, not a pointer
  2. somebody already wrote down the hash
  3. indistinguishable from a tampered dependency
  4. cached machines stay green, clean ones fail
  5. there is no un-publish, only forward

basics

~20 s

A published version is immutable by contract. Everyone who already fetched v1.3.0 recorded a hash of its content in go.sum, and the moved tag yields different bytes, so their build stops on a checksum mismatch. Fix it by publishing a new version.

solid answer

~50 s

Go treats a module version as permanently bound to the content it first had. The first time anyone fetched `v1.3.0`, the hash of that module's content went into their `go.sum` and the bytes went into their module cache; a mirror serving the module recorded them too. Force-moving the tag means the same version name now maps to different bytes, so a consumer fetching it fresh gets a hash that disagrees with the one in `go.sum` and the build stops with a checksum mismatch reported as a security error — which is correct, because that is exactly what a tampered dependency looks like. The symptom splits the team: whoever has the old content cached keeps building, whoever fetches fresh fails, and a mirror may keep serving the original. There is no un-publish; you fix forward by putting the intended content on v1.3.1 and moving consumers to it.

code

text · 7 lines
text
$ go build ./...
verifying github.com/acme/[email protected]: checksum mismatch
	downloaded: h1:...
	go.sum:     h1:...

SECURITY ERROR
This download does NOT match an earlier download recorded in go.sum.

go deeper

for a junior

Know one rule and hold it: never move or delete a tag you have already pushed. If a release is wrong, the next release fixes it.

for a middle

Explain the mechanism — a hash of the version's content was recorded on first use, so different bytes under the same version name fail comparison — and why a build that worked yesterday keeps working from cache.

for a senior

Show the diagnosis rather than the definition: reproduce on a clean machine with an empty module cache, explain why half the team is unaffected and why a mirror may still serve the original, and drive the fix to a new patch version instead of another force-push.

for a principal

Own the prevention. Decide who may cut releases, make automation refuse to overwrite an existing tag, and set the expectation that published versions are permanent — including the awkward consequence that a bad release stays fetchable and is superseded rather than withdrawn.

## Versions are immutable by contract The rule underneath everything here is that **a module version identifies content, permanently**. `v1.3.0` of your module is not "whatever the v1.3.0 tag points at today"; it is the bytes that were there the first time the world saw it. Go enforces this by recording a cryptographic hash of a module's content in the consumer's `go.sum` on first use, and comparing every later download against it. Moving a published tag breaks that binding. Nothing in Git stops you — `git tag -f v1.3.0 && git push --force origin v1.3.0` succeeds — but the ecosystem has already made a promise on your behalf. ### What consumers see ``` $ go build ./... verifying github.com/acme/[email protected]: checksum mismatch downloaded: h1:... go.sum: h1:... SECURITY ERROR This download does NOT match an earlier download recorded in go.sum. ``` The wording is deliberate. From the go command's point of view there is no difference between "the maintainer moved a tag" and "someone substituted a dependency's code". Both present as content that does not match a previously recorded hash for a version name. The only safe response is to refuse to build, loudly, and that is what happens. ### Why the team splits into two camps This is the part that makes the incident confusing before anyone identifies the cause. - An engineer who built yesterday has the original content **in their module cache** and a matching `go.sum` line. They do not refetch, so they see nothing wrong. Their CI, if it caches too, is also green. - An engineer on a clean checkout, or CI with a cold cache, fetches now. If they get the new bytes, `go.sum` disagrees and they fail hard. - A mirror that already downloaded and stored `v1.3.0` may keep serving the **original** content indefinitely, so the new commit's code may not even be what people receive — meaning the fix you thought you shipped never arrives. - A brand-new consumer with no `go.sum` entry has nothing to compare against, so they take whatever they are served and record it. Now two teams have different "v1.3.0" in their lockfiles. So the failure is not uniform, is not reproducible on the maintainer's machine, and does not correlate with anything in the consumer's own change. That is the signature to recognise. ### Diagnosing it The decisive test is a **clean machine with an empty module cache**: fetch the exact module at the exact version and inspect what arrives. Compare that against what a colleague who has been building all week has. If the same version name yields different content on two machines, the version has been mutated, and the repository's tag history (a tag pointing at a commit newer than the release) confirms it. `go.sum` lines from two branches of your own repository, one written before the move and one after, are the other tell. ### The fix, and the non-fixes **The fix is to publish forward.** Put the content you actually want on a new version — `v1.3.1` — push that tag, and tell consumers to upgrade. That is the whole remedy available to you as a maintainer. **Restoring the tag to the original commit helps, partially.** It makes fresh fetches match what the majority already recorded, so new failures stop. It does not help anyone who already recorded the new content in their `go.sum`; they now hold a hash nobody else will ever produce again and must reconcile. **Telling consumers to bypass verification is not a fix.** It asks every downstream team to disable an integrity check because of a mistake upstream, and it normalises exactly the behaviour that makes supply-chain compromise easy to miss. **Deleting the tag is worse than moving it**, because the version stops resolving for anyone whose cache goes cold, while remaining in their `go.mod`. ### The discipline that prevents it Once a tag is pushed, treat it as read-only forever. Cut releases from reviewed commits, never from a branch you intend to amend; if a release is wrong, the answer is always another release, and a patch version costs nothing. Make the release step in your automation refuse to overwrite an existing tag, because the only way this incident happens is that someone had the ability to force-push one.

  • Why does the go command call this a security error instead of just refetching?
    Because content that no longer matches a recorded hash for a version is precisely what a substituted or tampered dependency looks like, and the toolchain cannot distinguish a careless maintainer from an attacker. Failing loudly is the entire value of recording the hash in the first place; silently accepting new bytes would make the check decorative.
  • Two engineers on the same team see different results. What explains the split?
    Whoever already fetched the version holds the original content in their module cache and the original hash in go.sum, so nothing is refetched and nothing fails. Whoever builds from a clean checkout, or on CI with a cold cache, downloads now and hits the mismatch. A mirror may also still be serving the original bytes.
  • You force-moved the tag because the release shipped a bug. What should you have done?
    Commit the fix and publish v1.3.1, then tell consumers to upgrade. A patch release costs nothing and is the only mechanism that reaches people safely. The buggy v1.3.0 stays fetchable forever, which feels wrong but is the property that makes every consumer's build reproducible.
  • Does restoring the tag to its original commit undo the damage?
    It stops new failures, because fresh fetches once again match what most consumers recorded. It does not help anyone who fetched during the window and wrote the new content's hash into their go.sum — they hold a hash that will never be produced again and have to reconcile their lockfile by hand.

It is like changing what a page number refers to after the book has shipped: everyone who wrote down a quotation against that page is now, from the book's point of view, quoting something that does not exist.

saying these in an interview costs you the question

  • Says just force-push the tag back and it is fine
  • Advises consumers to disable checksum verification
  • Thinks a version tracks the latest commit on a branch
  • Assumes deleting the bad tag is cleaner than moving it
  • Concludes it is a consumer-side cache bug
  • Believes a version can be un-published