skip to content

A rebuild of the same source produces a different image digest — which build inputs typically differ between the two runs?

level: middleimportance: should knowfreq 48%

answer

  1. digest follows bytes, not intent
  2. the build recorded the run
  3. clock, file order, paths, base
  4. modification times change layer bytes
  5. a moving base reference re-resolves

basics

~20 s

Non-deterministic inputs the build recorded: embedded build timestamps, file modification times and archive entry ordering, absolute build paths and host names, and a base image whose reference resolved to different bytes. Each changes layer bytes, so the digest changes.

solid answer

~40 s

An image digest is derived from layer bytes, so a changed digest means some byte changed. With the source unchanged, the culprit is usually something the build captured about the *run* rather than about the code. Four families cover most cases: timestamps written into generated files and build history metadata; file modification times and the order entries are added to a layer archive; absolute paths and host names baked into compiled artifacts or generated config; and a base image named by a moving reference that now resolves to different bytes. The fixes are symmetrical — pin the base by digest, normalise file times and entry ordering when layers are assembled, strip or fix the build path, and keep facts about the run in a record published beside the image rather than inside a layer.

code

pseudocode · 16 lines
pseudocode
first  = build(source = revision, base = baseDigest)
second = build(source = revision, base = baseDigest)

if first.imageDigest == second.imageDigest:
    report "identical for this pair of runs"
    stop

shared = min(first.layers.length, second.layers.length)
for i in 0 .. shared - 1:
    if first.layers[i].digest != second.layers[i].digest:
        report "first divergent layer:", i
        diff entryNames(first.layers[i]), entryNames(second.layers[i])
        diff entryTimes(first.layers[i]), entryTimes(second.layers[i])
        stop

report "layers match; the difference is in the image document"

go deeper

for a junior

Remember that an image digest comes from the bytes of its layers, so anything a build writes differently between runs — even a timestamp — produces a different digest.

for a middle

Explain the concrete carriers: timestamps written into generated files and build metadata, file modification times and entry ordering inside a layer, absolute paths and host names, and a base that re-resolved.

for a senior

Show how you isolate it: build twice in a row, compare layer digests to find the first divergent layer, then diff metadata as well as content and remove or normalise the offending input.

for a principal

Weigh what determinism costs the build graph against what it buys — the ability to re-derive a release later — and decide which artifacts are worth holding to that standard.

## Why a digest changes at all An image is a set of read-only **layers** plus a document that lists them. Each layer is stored under a digest derived from its bytes, and the image's own digest is derived from the document naming those layers. **Content addressing has no notion of "the same source" or "the same intent"** — it sees bytes. So the question "why did the digest change?" always reduces to "which bytes changed?", and when the source did not change, the answer is an input the build captured about the run rather than about the code. A build reads far more than a source tree. It reads the clock. It reads modification times from the working copy. It walks directories in whatever order the filesystem hands them back. It knows its own absolute working directory and the machine's name. And it resolves whatever reference names the **base image** at the moment it runs. Any of those that ends up inside a layer makes that layer's bytes a property of the run — and the digest follows. ## The inputs that usually differ | Input | How it reaches the image | Why it varies between runs | |---|---|---| | Build timestamp | written into generated files, archive headers and build history metadata | the clock moved | | File modification times | stored per entry when a layer archive is assembled | a fresh working copy, or times set to "now" | | Entry ordering | the order files are added to a layer | directory iteration order is not guaranteed | | Absolute paths, host name | embedded in compiled artifacts, generated config, logs | the build ran in a different directory, on a different machine | | Base image bytes | every layer above inherits the change | a moving reference re-resolved to a newer image | Some notes on each: - **Timestamps** are the most common and the easiest to miss, because a build that stamps itself is doing something reasonable — it just does it inside the artifact. - **Modification times** are invisible to a content diff. Two layers whose files are byte-identical can still hash differently because the metadata stored alongside each entry differs. - **Entry ordering** is the subtle one: the same set of files, added in a different order, produces a different archive and therefore a different layer digest. - **Absolute paths and host names** leak through anything that records its own build environment — generated headers, debug information, configuration written at build time. - **The base** is the one that is not your build's fault. Everything above a changed base is different, so a single re-resolved reference can make every layer diverge at once. ## Pinning the base is a precondition A base named by a movable reference is resolved when the build runs. Months later the same name can legitimately point at different bytes, and your rebuild starts from a different floor. Naming the base by **digest** removes exactly one variable: the rebuild begins from the same bytes it began from the first time. It does not make the rest of the build deterministic — your own steps can still stamp the clock into a file — which is why it is a precondition rather than a solution. ## Isolating the culprit 1. Build twice **in a row on one machine** from the same source revision and the same pinned base. If those two already differ, the non-determinism is inside your build and you never needed the year-old release to find it. 2. Compare layer digests in order and stop at the **first** divergent layer — everything above it is downstream noise. 3. Unpack both copies of that layer and diff **metadata as well as content**: entry names, entry order, modification times, permissions. 4. Map what differs back to the step that produced it, and remove or normalise the input. 5. Re-run the pair and confirm the two digests now agree before moving up the stack. ## What a fix looks like - Pin the base by digest. - Normalise modification times to a fixed value derived from the source revision, rather than to the wall clock. - Sort entries when assembling a layer so ordering is a function of the file set, not of the filesystem. - Build from a path that is the same everywhere, or strip the path from anything that records it. - Move the run's own facts — when it started, on what machine, with which parameters — into a record **published beside the image**, keyed to its digest, instead of into a layer. Nothing is lost; the bytes simply stop encoding the run. ## What this is not Making a build deterministic is not the same as constraining what inputs it may reach for — declaring every input up front and cutting the build off from the network is a separate discipline with its own owner, and a build can be fully deterministic without it. Determinism also says nothing about whether the artifact is any good. It buys one specific thing: the ability to produce the same bytes twice, which is what makes a later comparison mean something.

  • What does pinning the base by digest change about a rebuild months later?
    A base named by a movable reference is resolved at build time, so a second run can start from different bytes and every layer above it differs. A digest names exactly one set of bytes, so the rebuild starts from the same floor. It is a precondition, not a cure: pinning the base removes the input you cannot see changing, but your own steps can still stamp the run into a layer.
  • How do you keep an honest record of when and where a build ran if those facts must not go inside a layer?
    Publish them beside the image rather than inside it. The run's clock time, machine and parameters belong in a record the build emits alongside the artifact and keys to its digest, while the layers themselves get normalised times and ordering. Nothing is hidden — the facts are still available to anyone who wants them — the bytes just stop encoding a particular run.

saying these in an interview costs you the question

  • Claims two builds of one revision always produce one digest
  • Says file timestamps are metadata that digests exclude
  • Blames the registry for rewriting layers during upload
  • Treats a version-like tag on the base as a pin
  • Thinks building on the same machine makes a build reproducible