Every helm upgrade of a chart rolls all its Pods although the image and values are unchanged - how do you find the cause?

## The scenario A fraud-scoring endpoint runs at 6 replicas from a chart that also carries a pre-upgrade migration hook and a 240-line values file. The team ships small chart edits constantly - roughly eleven version bumps a week, mostly `2.14.3` to `2.14.4` style patch bumps for template tidying. Every one of those upgrades restarts all 6 scoring Pods, each of which loads a model on start-up. Nobody changed the image, and nobody changed a value. ## Step 1: prove it is the Pod template Do not diff the whole manifest - it always differs, because the version labels are on every object. Compare only what the controller actually watches: ```bash helm get manifest scoring-prod --revision 41 > /tmp/r41.yaml helm get manifest scoring-prod --revision 42 > /tmp/r42.yaml diff /tmp/r41.yaml /tmp/r42.yaml ``` and read only the hunks inside a `spec.template` block. Everything else is noise for this question. To predict the *next* upgrade before running it, render the pending change with `helm upgrade --dry-run=server` and diff that against the current stored manifest. Do not be misled by the migration hook: hook Pods are created fresh for each upgrade by design, so a new Job Pod every time is expected and is not the workload restarting. ## Step 2: the three usual causes **Version-bearing labels on the Pod template.** This is the most common by a wide margin. The chart's full label set - the one that correctly stays out of the immutable selector - is applied to `spec.template.metadata.labels`, and it contains `helm.sh/chart: scoring-2.14.4` and `app.kubernetes.io/version`. Bump either version and the Pod template is different, so the workload rolls. Nothing is broken; the chart is doing what it was written to do. The question is whether you meant it. **A checksum annotation with too broad an input.** A chart that writes `checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}` hashes the *entire rendered file*, and that file's own `metadata.labels` carry the same version-bearing labels. So the digest changes on a version-only bump even though not one config key moved. Narrowing the input to the values subtree that actually feeds the config - `{{ toYaml .Values.scoring.config | sha256sum }}` - removes the coupling and rolls Pods exactly when configuration changes. **A non-deterministic render.** Anything that returns a new value on each render and reaches the Pod template, directly or through something you hash: a timestamp, a random string, a freshly generated certificate or password. The tell is that two consecutive `helm template` runs with identical inputs already differ - which is the cheapest test in the whole investigation, because it needs no cluster at all. Render twice, diff, and if the local renders disagree the cause is in the chart, not the upgrade. ## Step 3: decide, do not just fix Once you know which field moves, this becomes a judgement call about what a restart is worth. - **Keep the version labels on the Pod template** and accept the rolls. You get per-Pod version attribution: a query over running Pods answers "what is deployed" without consulting release history. For a workload where a restart is cheap this is the right default. - **Emit the version labels on the object's metadata only**, and keep the Pod template's labels down to the stable ones. Rollouts then track image and config changes rather than chart edits. The cost is that Pods no longer carry the app version, so dashboards and log queries that group by it lose their key - check what consumes that label before you remove it. - **Narrow the checksum input.** This one is close to free and worth doing regardless. - **Reduce the churn at its source** if the eleven bumps a week are mostly cosmetic - but that is a release-hygiene answer, and the chart should still be correct under a high bump rate. For a model-loading scoring endpoint I would narrow the checksum, strip the chart and version labels from the Pod template while keeping them on the object, and note the decision in the chart's documentation so the next author does not "restore" them. ## What not to do Do not reach for a replacement flag - the resource is applying fine, and replacing it makes the disruption total instead of rolling. Do not add a timestamp annotation to make rollouts "predictable": that guarantees a restart on every upgrade, which is the problem, not the fix. And do not conclude the platform is restarting Pods on its own; the Pod template changed, and the diff will show you exactly where.