Two helm package runs over identical chart source produce .tgz files with different digests. Why?

### Why the digests differ at all People reason about a chart archive as if it were content-addressed: same files in, same hash out. It is not. `helm package` writes a tar archive and compresses it, and a tar archive stores, for every entry, a header containing the path plus file metadata — modification time among it. Two builds of the same commit on two machines, or on the same machine after a fresh clone, produce files whose mtimes are the moment the checkout happened. The chart content is identical; the tar bytes are not; the `sha256` differs. This surfaces the moment anyone starts comparing artifacts. A pipeline builds `pdfsign-2.7.3.tgz` in the build job and again in a verification job, and gets `sha256:9c41…` and `sha256:2be7…` for what is unmistakably the same chart. Someone concludes the source is not what they think it is, or that something in the pipeline is tampering. Nothing is wrong; the format simply records more than the content. ### The fix Helm 4 provides Helm 4's `helm package` honours `SOURCE_DATE_EPOCH`, the cross-ecosystem convention for reproducible builds: an environment variable holding a Unix timestamp in seconds, which build tools use in place of "now" wherever they would otherwise embed a time. Set it — conventionally to the timestamp of the commit being packaged — and the archive's entry timestamps are pinned to that value, so packaging the same source produces the same bytes on any machine at any time. ```bash export SOURCE_DATE_EPOCH=$(git log -1 --format=%ct) helm package -d ./dist ./pdfsign sha256sum ./dist/pdfsign-2.7.3.tgz ``` Run that twice from two clean clones of the same commit and the digests match. Helm 3.21 does not implement this; on Helm 3 the practical answer is that the artifact is built once, stored, and referenced by digest thereafter — you compare stored copies, not rebuilds. ### What still moves after you pin the clock Timestamps are usually the whole explanation, but a reproducibility claim is only as strong as its weakest input, and packaging has three more: - **Dependency resolution.** `-u/--dependency-update` re-resolves what `Chart.yaml` declares and downloads into `charts/`. If a dependency is declared with a range rather than an exact version, a build a week later legitimately vendors a different subchart and the archive genuinely differs. For a chart with an optional bundled database subchart this is the common case. Build from a committed lock or from vendored `charts/` when you want byte-equality, and treat a floating dependency range as an explicit decision to be non-reproducible. - **Metadata stamping.** `--version` and `--app-version` change `Chart.yaml` inside the archive. A pipeline that stamps a build counter into the version is producing a different artifact every run by design; that is fine, but it is not reproducibility. - **The file set.** Anything `.helmignore` lets through that varies between checkouts — a generated file, a local override someone's environment writes — changes the archive. Reproducibility failures that survive a pinned `SOURCE_DATE_EPOCH` are almost always this: something in the tree is not in version control. ### Why anyone cares Three reasons, in increasing order of seriousness. First, caching and comparison. If a rebuild yields the same bytes, a pipeline can skip republishing, and a reviewer can confirm that the artifact on the shelf corresponds to the commit it claims. Second, incident work. When you are asked whether the chart running in production is the chart in the tag, a reproducible build turns a debate into a command. Without it, you can only compare the extracted contents, which is doable but not something you want to be inventing at 2 a.m. Third, verification generally. Digests are how artifacts are referenced once they leave your machine — a repository index records one, and any verification step downstream is checking one. An artifact whose digest changes every build is one that no downstream check can pin, and it pushes teams toward the weaker habit of trusting a version *string* instead. Verifying an archive's authorship is a separate mechanism from making its bytes stable, but the second makes the first far more useful. ### The pragmatic position Set `SOURCE_DATE_EPOCH` from the commit in every chart-packaging job — it is one line and it removes a whole category of confusing pipeline noise. Pin dependency versions rather than ranges. Then, when two builds still disagree, you have a real finding rather than a shrug, and the diff of the extracted trees will tell you exactly which file is not coming from the repository.