helm install --verify fails on a chart that installs fine without it. How do you diagnose it?

`--verify` is a small surface, which is what makes this triage tractable: it can only fail for a handful of reasons, and each has a distinct signal. The scenario worth holding in mind is the ordinary one — an internal `maildigest` chart (the email-digest builder, version 2.14.3, a 240-line values file and a `values.schema.json` contract) that installs cleanly in a developer shell and fails in the pipeline the moment `--verify` is added. ### Step 0: separate the flag from everything else `helm install` without `--verify` proves nothing about provenance — it means the chart resolved and rendered. So do not start by comparing the two commands; start by removing the cluster entirely: ```bash helm pull internal/maildigest --version 2.14.3 --prov ls maildigest-2.14.3.tgz* helm verify maildigest-2.14.3.tgz --keyring /etc/helm/trusted-signers.gpg ``` If `helm verify` fails on the local pair, this has nothing to do with the cluster, the namespace, RBAC, or values. If the `.prov` never landed, you already have your answer. ### Cause 1: there is no provenance file For a chart resolved from a repository, `--verify` needs the server to host `<chart>-<version>.tgz.prov` beside the tarball. Publishers who ran `helm package` without `--sign` produce no such file; mirroring jobs frequently copy `*.tgz` and silently drop the sidecar; a repository that regenerates its index from tarballs it found on disk will happily list a chart whose provenance was never copied. The tell is that the file simply 404s. Fix it at the publisher: sign at package time and publish both artifacts as a unit, never one without the other. ### Cause 2: the key is not where Helm is looking `--keyring` defaults to `~/.gnupg/pubring.gpg`. A CI runner usually has no GnuPG home at all, so the default resolves to a nonexistent file and every signer is unknown. Even when a keyring exists, modern GnuPG keeps public keys in a keybox database rather than the legacy binary keyring Helm reads, so pointing at the wrong file gives you an empty trust set that looks identical to "signed by a stranger". The distinguishing evidence is that the failure is about the *signer or signature*, and it reproduces with a keyring you know is empty. The fix is to stop depending on ambient GnuPG state: build a keyring file containing exactly the public keys your organisation trusts, ship it to runners as a managed artifact, and pass `--keyring` explicitly everywhere. That also makes the trust set reviewable, which the developer's personal keyring never is. ### Cause 3: the digest does not match This is the interesting one, and the most common in an estate that mirrors charts. The `.prov` pins the SHA-256 of the exact tarball produced at signing. Anything that re-creates the archive — a mirror that unpacks and re-packs, a second `helm package` run publishing new bytes alongside the old `.prov`, a sync tool that "normalises" artifacts, a partially transferred file — changes those bytes and breaks the comparison while the signature over the document remains perfectly valid. You can confirm this by hand in seconds, because the `.prov` is clear-signed text: ```bash sed -n '/^files:/,+2p' maildigest-2.14.3.tgz.prov sha256sum maildigest-2.14.3.tgz ``` Two digests, one comparison. If they differ, no key or keyring change will ever help; the artifact you have is not the artifact that was signed. Fix the pipeline so the signed bytes are the published bytes — and if you need repeatable rebuilds, Helm 4's `helm package` honours `SOURCE_DATE_EPOCH` to keep tarballs reproducible. ### Cause 4, occasionally: mismatched pair A `.prov` from a neighbouring version can end up beside the wrong tarball after a hand-copy. The metadata block in the signed body names the chart and version, so reading the file tells you immediately that you are holding provenance for `2.14.2` next to a `2.14.3` tarball. ### What good triage looks like State the three buckets before touching anything, then collapse them with one command each: does the `.prov` exist, does the keyring contain the signer, does the digest match. Fix the cause at the layer that produced it — publisher, runner configuration, or mirror — rather than the layer that reported it. And say the thing an interviewer is listening for: dropping `--verify` to unblock the pipeline converts a genuine trust check into a comment in a runbook, so the flag is not the thing to remove while you investigate.