How would you make Helm chart provenance verification actually enforced across an estate installing from an internal chart repository?

This is a governance question wearing a flag's clothing. The Helm facts are small; the design space around them is where the answer lives. ### The constraint that shapes everything Helm is a client. There is no in-cluster Helm component that could refuse an unverified chart, and `--verify` is one optional argument among many. Anyone with cluster credentials and a terminal installs whatever they like. So "we require signed charts" phrased as "everyone passes `--verify`" is an aspiration, not a control — the first thing to say out loud. The second fact is ecosystem-shaped: chart signing is opt-in and most published charts have no `.prov` at all. A policy of "verify everything against its upstream publisher" would block most of the catalogue on day one. Any credible design has to say what happens to the unsigned majority. ### Design: verify at ingress, re-sign, distribute internally The shape that survives contact with reality is a single doorway. One job — the only identity allowed to publish into the internal repository — does the following for each external chart: 1. Pull the chart and, where the publisher provides one, its `.prov`, and run `helm verify` against a keyring of upstream signers you have chosen to trust. 2. Record what was admitted: chart name, version, and the SHA-256 of the exact tarball. 3. Package or re-publish those exact bytes and sign them with the organisation's key (`helm package --sign`, or signing the artifact you admitted without repacking it). 4. Publish `.tgz` and `.prov` together, always as a unit. Now internal consumers verify against **one** trusted key. That is the real win: the trust decision moves from "every team curates a keyring of vendor keys" to "the platform curates one key, and every install checks it". The honesty clause matters in an interview. Your organisation's signature on a chart you did not author attests *this artifact passed our ingress and these are the bytes we admitted*. It does not assert vendor authorship, and it says nothing about whether the chart's templates are safe. Say so; conflating the two is how signing turns into theatre. ### Key custody and rotation The private key is the whole system. Keep it in a managed key store or hardware-backed key that cannot be exported, usable only by the packaging job's identity, with a passphrase supplied at run time and never present in a repository. Log every signing operation. Then plan for the day it changes. Rotation is not free: consumers verify against a keyring file, so a new key must reach every runner, every developer machine and every image that runs installs before charts signed with it can be verified — and you have to decide whether to re-sign the existing catalogue or accept both keys during a transition window. Compromise is the same mechanics under time pressure. A design that cannot answer "how do we rotate" is a design that will never rotate. Distribute the keyring as a versioned, reviewed artifact rather than letting it accumulate in personal GnuPG homes. Helm reads the legacy binary keyring format, so publishing a prepared keyring file also removes a whole class of "it works on my laptop" failures. ### Know where your installs actually happen If a GitOps reconciler installs charts into your clusters, the `helm` CLI's `--verify` flag is not in the loop, and no amount of pipeline discipline puts it there. In that model, verification has to happen where the artifact enters the estate — which is another argument for the ingress design — and any additional at-apply enforcement is the job of an admission policy engine, a different system with a different owner. The same logic applies to a developer who runs `helm install` by hand: the only thing that reaches them is what is in the internal repository, so make what is in there the thing you have already verified. ### Coverage and its edges One digest covers the entire tarball: templates, `values.yaml`, a `values.schema.json` contract, `crds/`, vendored subcharts. It does not cover the container images the chart references — those are separate artifacts with their own publication path — and it does not cover the values a caller supplies at install time. Do not let a signed chart be presented as a signed deployment. Also watch the mechanical trap that will otherwise generate a steady stream of incidents: provenance pins bytes, so any mirroring or normalisation step that re-packages a tarball invalidates it. The ingress job must publish precisely the bytes it signed. ### Measuring it Ask the uncomfortable question: how would you know if verification stopped happening? If the answer is "the flag is in the pipeline template", note that pipeline templates get edited. Prefer controls whose absence is visible — publication into the internal repository being possible only through the signing job, and periodic re-verification of the repository's contents against your key, so an artifact that appeared some other way stands out. ### The tradeoff to name All of this costs a real doorway with an owner, an on-call story when the doorway is down, and latency between an upstream release and its availability internally. The alternative — verify opportunistically where signatures exist, enforce nothing — costs nothing and buys nearly nothing. Pick deliberately, and size it to whether your threat model is a compromised mirror and accidental tampering (which this stops) or a malicious chart author (which it does not).