After a Helm upgrade, one tenant's worker Deployment lost its entire tolerations block while other tenants kept theirs — how do you diagnose it?

### The shape of the incident A multi-tenant chart is installed once per team namespace, and each tenant supplies its own overrides — in the case that bites, a 240-line values file. After a routine upgrade, the invoice-rendering worker in one namespace comes back scheduled on the wrong nodes, and its Deployment has no `tolerations` field at all. Six other tenants upgraded from the same chart version in the same window and are fine. Nothing failed: the upgrade reported success, and there is no error to read. That combination — one tenant, a whole field gone, no error — is the signature of a template guard that collapsed for that tenant's values. ### Separate the two questions Ask them in this order, because they have different answers and mixing them wastes time. **Why did the render change?** The manifest Helm stored with each revision is the record of what the chart actually produced, so retrieve it for the last good revision and for the new one and compare: ```bash helm get manifest invoicing -n team-ledger --revision 37 > before.yaml helm get manifest invoicing -n team-ledger --revision 38 > after.yaml diff before.yaml after.yaml ``` Read the shape of the difference carefully. `tolerations: []` in the new manifest is a different bug from no `tolerations:` key at all. An absent key means the whole block was skipped, and in a chart that block is almost always written as a guard whose body carries the key itself: ```yaml {{- with .Values.worker.tolerations }} tolerations: {{- toYaml . | nindent 8 }} {{- end }} ``` Because the `tolerations:` literal lives inside the body, a falsy value takes both lines with it. **Why is it falsy for this tenant?** Compare the guarded path in the template with the values that release actually stored. In a 240-line file the usual causes are all indentation-shaped: the tenant's key sits under a sibling of `worker` rather than under `worker`, so the path resolves to nil; or the chart moved the key in this version and the tenant's file still writes the old location; or an override sets it explicitly to `null` or to an empty list. Every one of these is falsy, and falsy is indistinguishable from deliberately unset. Confirm by rendering the same chart version against that tenant's values and watching the block reappear when the key is placed correctly. ### Why the live object lost the field rather than keeping it This is the part people find surprising: the previous Deployment had the field, and Helm was not asked to remove it. But an upgrade is not an additive patch. Helm compares what it is sending now with what it sent before and withdraws what it no longer declares, so a field that has dropped out of the render is genuinely retracted from the live object. Under Helm 4's default server-side apply path this happens through field ownership — Helm's field manager owned the key, stopped sending it, and the API server removed it. Under the older client-side three-way merge that Helm 3 uses, and that a release first installed by Helm 3 keeps on upgrade because `--server-side` on upgrade defaults to inheriting the previous method, the same conclusion is reached by diffing the stored manifest against the new one. Different mechanism, same outcome: an omitted field is a removal, not a no-op. The corollary is worth stating out loud in a post-mortem — a values typo in an optional block is not a cosmetic defect, it is an unrequested change to a running workload. ### Stopping the silence The defect class is that an empty value and a missing value are the same thing to a guard. Three fixes attack it at different levels. Constrain the input. A chart's values schema can reject a values file that lacks a required key or that gives it the wrong type, which turns a mis-nested override into a failed upgrade instead of a silently missing field. That is the cheapest guard against the indentation family of mistakes. Make the field non-optional in the template where the field is not optional in reality. If the worker must always carry scheduling constraints, do not wrap them in a guard at all: render them from a value that has a real default, so an override can change them but nothing can make them vanish. Diff before you ship. For a chart installed once per tenant, rendering each tenant's values and diffing against the manifest of the current revision catches exactly this class before an upgrade runs, because the disappearance is obvious in a diff and invisible in a success message. ### What good judgement looks like here The instinct to resist is patching the one tenant's values and moving on. The interesting question is why a single misplaced line in one of seven values files could silently remove a scheduling constraint from a production workload, and what in the chart's contract should have refused it.