Why does a Helm chart that generates a password with randAlphaNum break on upgrade?

### Templates have no memory A Helm chart is a pure function of its inputs. Every `helm upgrade` re-renders all of `templates/` from scratch against the merged values, and nothing in that render is carried over from the previous revision. `randAlphaNum 24` is sprig's random-string generator: it produces a fresh 24-character string every single time it is evaluated. So a template like ```yaml data: passphrase: {{ randAlphaNum 24 | b64enc | quote }} ``` produces one passphrase at install and a *different* passphrase at every subsequent upgrade, including upgrades that change nothing else. Helm then applies that Secret, and the credential the cluster holds silently stops matching the credential anything else knows. ### What the failure looks like in practice For a PDF-signing service whose passphrase unlocks a key stored elsewhere, the symptom is that signing starts failing shortly after an unrelated chart bump — a resource-limit tweak, say — with an authentication or decryption error and no deployment that looks relevant. Two details make it confusing: * If the workload only reads the Secret at process start, the running pods keep working on the old value until something restarts them, so the failure can appear hours later on the next node drain, and correlating it with the upgrade is hard. * If the chart carries a checksum annotation over the rendered config — the common `checksum/config` pattern that rolls pods when their configuration changes — the annotation changes on *every* upgrade, because the random value changed. Now every upgrade is a full rollout even when nothing was edited, and the breakage is immediate rather than delayed. Chart churn like that is often the first clue that something in the render is non-deterministic. Either way, the diagnosis is the same: render the chart twice and diff it. `helm template` run back to back against identical values should be byte-identical; if the Secret differs between two runs, the chart is generating the value, not carrying it. ### The three real fixes **1. Make it an input.** The simplest and most defensible option: the chart does not generate the credential at all. It takes it from values (or, better, takes the *name* of a Secret something else created) and refuses to install without it, using `required` to fail the render with a readable message rather than shipping an empty string. Determinism is restored because there is nothing random left. **2. Look up the existing value and reuse it.** Helm's `lookup` function reads a live object from the cluster at render time, so the template can keep whatever it generated on the first install: ```yaml {{- $name := printf "%s-signing" (include "pdfsign.fullname" .) -}} {{- $prev := lookup "v1" "Secret" .Release.Namespace $name -}} {{- $pass := "" -}} {{- if $prev -}} {{- $pass = index $prev.data "passphrase" | b64dec -}} {{- else -}} {{- $pass = randAlphaNum 24 -}} {{- end }} ``` This works, and it is what mature charts do — but be honest about its caveats, because an interviewer will push on them. `lookup` needs an actual cluster connection and returns an empty result during a client-side render, so `helm template` and a client-side dry run take the *generate* branch and produce a different answer every time; any pipeline that diffs rendered output has to account for that. It also means the chart's output now depends on cluster state, so the same values no longer determine the result. And on a restore into a fresh namespace the lookup finds nothing and a brand-new credential is generated, which is a silent data-loss scenario if something outside the cluster trusts the old one. **3. Move ownership out of the chart.** Let something outside the release create and rotate the Secret — a bootstrap step, a platform process, or an external secret operator — and have the chart consume it by name. Helm then never renders the material, never stores it in the release record, and cannot clobber it. ### The fix that does not work The frequent wrong answer is `helm.sh/resource-policy: keep` on the generated Secret. That annotation controls **deletion**: it keeps the object when the release is uninstalled or when the resource is dropped from the chart. It does not exempt anything from being updated. An upgrade still writes the newly generated value straight over the existing one, so the annotation buys nothing here — and it also leaves an orphaned Secret behind after uninstall, so a later reinstall inherits an object Helm no longer thinks it owns. The second wrong answer is "pin it with a pre-install hook so it only runs once". A `pre-install` hook genuinely does not run on upgrade, so a hook-annotated Secret is created once — but hook resources are outside the release's normal lifecycle and are commonly deleted by a `hook-delete-policy`, so this trades a regeneration bug for an ownership one. Use `lookup`, or stop generating.