How far should a team chase bit-identical container image rebuilds versus an auditable rebuild?

### Two different properties wearing one word **Bit-identical rebuild**: give an independent party the same source and the same declared inputs, and their build produces byte-for-byte the same image - the same digest. It is a *verifiable* property. Anyone can check it without trusting your pipeline, because the check is a hash comparison. **Auditable rebuild**: for any image running in production, you can state exactly what went into it - which commit, which base image digest, which dependency lock, which builder - and rebuild something functionally equivalent, without claiming the bytes match. It is an *attestable* property: it relies on records your pipeline kept, so it is only as good as your trust in that pipeline. They answer different questions. Bit-identity answers "can someone who does not trust us confirm this image came from that source?". Auditability answers "when this thing misbehaves, do we know what is inside it and can we get back to it?". The second question is the one most teams are actually being asked, and it is far cheaper to answer. ### The cost curve The first 80% is nearly free and pays for itself immediately: pin the base image by digest, resolve dependencies from a lockfile, set `SOURCE_DATE_EPOCH` from the commit, stop stamping build dates into labels, record the resulting image digest against the commit. That combination gets most images to "rebuilds usually match", and more importantly it makes any remaining difference *informative* rather than noise. The next 15% costs real engineering: OS packages pinned against a snapshot of the distribution archive or against a mirror you run, downloads replaced by vetted artefacts copied from the context, builds that touch no unmediated network. The last 5% is where budgets die: a compiler that embeds absolute paths or parallel-build nondeterminism, an archiver whose entry order varies, a vendor base image that is not itself reproducible, a build tool that mints an identifier. This tail is frequently owned by somebody else entirely, and chasing it can consume a quarter with nothing shipping. ### How to decide Ask what the property is *for*, per class of image: - **Independently verifiable artefacts** - something you publish for others to run, or something that must satisfy a customer or regulator that the binary matches reviewed source. Bit-identity earns its cost here, because the value is precisely that a third party can check it without trusting you. - **Internal services** - the fraud-scoring service and its 2.3 GB dependency layer, deployed only by you. What you need in an incident is to know what is inside the running image and to be able to reconstruct an equivalent one. Auditable is enough; spend the effort on pinning and on recording digests, not on chasing the last byte. - **Base images you publish internally** - worth more rigour than the services on top of them, because every service inherits their inputs, and pinning downstream is only meaningful if the thing being pinned is itself well-defined. ### Making it stick organisationally A policy that is not measured decays. The practical shape is a rebuild-and-compare canary: for a small set of images, rebuild the released commit on a cold cache, compare the image ID against the one recorded at release, and alert on a mismatch. Keep the set small enough that the signal stays trustworthy - a check that goes red every week for a reason nobody fixes teaches the whole team to ignore it, which is worse than not having it. Two failure modes to name in an interview, because they show operational judgement: 1. **Reproducibility turning into staleness.** Digest pins freeze security patches. Whatever you mandate, someone or something must be accountable for moving pins on a schedule; otherwise the estate becomes reproducibly vulnerable. 2. **Optimising the wrong property.** A team can achieve bit-identical rebuilds and still not know what is inside its images, because bit-identity says the build is a function of its inputs - it says nothing about whether those inputs were reviewed. ### What a strong answer sounds like "Bit-identity is a verification property and we need it only where someone outside the team has to check us; for our own services, the goal is that we can say what is in an image and rebuild an equivalent one. So: digest-pinned bases and lockfiles everywhere, normalised timestamps because they are free, recorded digests per release, and a rebuild-diff canary on the handful of images where it matters. I would not fund chasing toolchain nondeterminism across the estate until something concrete depends on it." That is a position, with a boundary, and with an owner for the pins - which is what the question is testing.