After an internal library's version 2.4.1 was re-published with different contents and two services built different code from it, how do you prove it and prevent a repeat?
answer
- compare hashes recorded earlier
- sidecar checksum moved too
- which cache kept the old bytes
- refuse redeploy on releases
- ship 2.4.2, never re-fix 2.4.1
basics
~20 sCompare the hashes each service recorded at build time with what the repository serves now, and find the second upload in its history. Then make release repositories refuse redeploys and ship the fix as a new version.
solid answer
~50 sFirst prove two contents existed: take the hash each build recorded earlier, such as an npm lockfile's `integrity`, a checksum in the build log or the file in a runner's cache, and compare it with the bytes the repository serves now. The `.sha1` beside a Maven file proves nothing here, because the redeploy uploaded a fresh checksum with the new file. The repository manager's upload history shows when the path was written twice and by which identity. Then explain the split: one build resolved through a cache, such as a proxy or a runner's local repository, that kept the first copy. To prevent a repeat, configure every release repository to refuse a second upload to an existing path, restrict release writes to the CI identity, and publish the fix as 2.4.2. Do not re-upload the original bytes to 2.4.1; caches would then hold a third state.
go deeper
Recall that a release version must never be uploaded twice and that a broken release is fixed with a new version number.
Explain why caches held the first upload and why the checksum file beside an artifact changes along with it.
Walk the diagnosis: hashes recorded elsewhere, the repository's upload history, each service's resolution path, then containment with 2.4.2 and a release repository that refuses redeploys.
Turn the incident into an estate-wide audit: which repositories and registries accept overwrites today, who may write release paths, and how the guarantee is verified continuously.
## The incident Version `2.4.1` of an internal library was uploaded twice to the same release path with different contents. Service A built against the first upload and service B against the second. Both build logs say "2.4.1". The tasks are to **prove** what happened, **explain** why the two services saw different bytes, and **stop** it recurring. This is a diagnosis with one right answer once the facts are in, which is what makes it a production question rather than a design debate. ## Step 1: prove that two contents existed The evidence has to come from somewhere the redeploy did not touch: - **Hashes recorded at build time.** An npm lockfile stores each package's `integrity` hash; many builds log the checksum or digest of what they resolved; a CI runner's local cache may still hold the original file. - **The bytes served now.** Download the current file from the release repository and hash it. - **The repository manager's own record**: the path's created and modified times, the deploying identity, and the request log, if the product keeps one. What does **not** prove anything is the checksum file stored next to the artifact. Most Maven repositories keep `.sha1` and `.md5` files beside each file, and the deploy that replaced the file uploaded matching new ones. Maven Resolver's documentation says checksums retrieved alongside an artifact only provide integrity verification; they do not establish that the checksum itself is trustworthy. A checksum proves "unchanged" only when it was recorded earlier and kept elsewhere. ## Step 2: explain the split Two services see different bytes under one version because **copies of the first upload survived**: 1. A **proxy repository** at another site, or one fronting a replicated copy, cached the first file. Release files are treated as fixed and are usually served from the cache rather than re-fetched; in Artifactory, for example, zapping a remote repository's cache expires cached metadata but does not invalidate cached binaries. 2. A **CI runner or developer machine** had the file in its local cache and never asked the server again. 3. A clean runner, or a site without the cached copy, fetched the second upload. A Nexus Repository group searches its members in list order, and an Artifactory virtual repository resolves in a configured order too, so different results are not random: trace each service's resolution path to the cache that answered it. ## Step 3: contain it 1. **Pick the canonical content.** Decide which build is the one that should have shipped, usually the one already in production. 2. **Publish a new version**, `2.4.2`, with that content or with the intended fix, and move both services to it. 3. **Do not re-upload anything to 2.4.1.** Restoring the "original" bytes is another overwrite: caches that took the second upload would now disagree with the server too, giving a third state. 4. **Mark 2.4.1 as not to be used**, using whatever the ecosystem offers. ## Step 4: prevent a repeat | Control | Where | Effect | |---|---|---| | Release repository refuses a second upload to an existing path | Repository manager, per release repository | The redeploy fails instead of succeeding quietly | | Separate release and snapshot repositories (version policy, or Handle Releases and Handle Snapshots) | Repository manager | Rebuilds of a development version can never land on a release path | | Only the CI release identity may write release paths | Repository permissions | No laptop or ad hoc job can publish a release | | Immutable tags for version-shaped container tags | Registry setting, such as Harbor's rules | The same failure cannot happen to images | | Consumers record and check hashes | Lockfiles, deployment records | Any future mismatch is caught at install time | The server-side refusal is the control that matters most. The others narrow who can trigger the problem or make it visible, but only the repository can make the second upload impossible. ## Why the timing hides it An overwrite like this usually surfaces days later, not at the moment of the second upload. The build that pushed the second copy succeeds, and the services that already cached the first copy keep working. The symptom appears only when a clean environment, such as a new CI runner or a new site, builds the same version and behaves differently. That delay is why the hash recorded at build time matters: it is the only witness from before the change. ## What to say in the review The root cause is not the person who re-ran the publish job; it is a release repository that accepted a second write to an existing release path. Every repository and registry that holds releases should be audited for the same gap, because defaults differ between products and between formats.
- Why not just clear every cache so both services fetch the current 2.4.1?That makes the services agree but leaves the cause in place: the release repository still accepts overwrites, and every copy of the first upload outside your control, such as other teams' runners and remote sites, still disagrees. It also silently changes what a past build would reproduce. A new version fixes the naming; a refusing release repository fixes the cause.
- What would have caught the mismatch at build time instead of in production?A hash recorded when the version was first adopted and compared on later installs, such as the integrity string an npm lockfile stores for each package. A changed file under the same version then shows up as a mismatch at install time instead of building quietly. A checksum served beside the file cannot do this, because it changes with the file.
- The .sha1 file next to the artifact proves the release never changed.
- Re-uploading the original bytes to 2.4.1 cleanly undoes the incident.
- Clearing every cache fixes it, so no repository setting needs to change.
- A group repository returns members' copies at random, so the split cannot be traced.
- The person who re-ran the publish job is the root cause.