Across 400+ Helm releases in one cluster, how do you decide storage driver, retention and recovery for release state?

### Frame it: derived state or system of record? Every other decision follows from this one. Helm's release records hold the chart, the values and the rendered manifest for each revision. If the chart version and the values that produced a release are reproducible from version control, the records are a *cache* — convenient, load-bearing for rollback, and re-creatable in principle. If people deploy with ad-hoc `--set` flags from laptops, the records are the only place the deployed configuration exists, and you are running an unbacked-up system of record inside the cluster it manages. Most estates are somewhere in between and have never said which. Saying it out loud changes the recovery plan, the retention number and how much you care about backups, so it is the first thing to settle. ### Backend The default keeps a Secret per revision in the release's own namespace. The properties that gives you are worth naming, because they are why the default usually wins: state is namespaced, so tenant boundaries apply to it automatically; it inherits the namespace's RBAC and whatever the cluster does to protect Secrets; it is included in any backup that covers the cluster; and there is no second system that can be unavailable at the moment you need to roll back. Moving to `sql` is justified by exactly two constraints — a chart whose record does not fit in one object, and record volume that is a real load on the cluster datastore. At 400 releases with default retention you are talking about a few thousand objects; sizes vary hugely, so measure before assuming. What you pay is real: release state now lives outside the cluster it describes, so a cluster restore no longer restores release history; every install, upgrade and rollback depends on a reachable database; the connection string is a credential every deploying identity holds; and your tenant isolation is no longer namespace-shaped. I would exhaust cheaper levers first — retention, and shrinking what charts carry — and adopt `sql` only for the specific estate that genuinely needs it, not fleet-wide because one chart is large. The corollary is discipline about the variable: because the backend is read from the environment per invocation, a non-default choice must live in a shared wrapper or base image. A split where CI uses one backend and engineers use another produces two groups who disagree about which releases exist, and an `upgrade --install` that tries a fresh install over live objects. ### Retention Retention is a rollback horizon expressed as a count, and it should be derived from a time. Ask how long a bad change typically survives before someone notices — the answer for a telemetry ingest gateway that deploys twelve times a day is very different from a batch service that deploys monthly. Multiply detection time by deploy rate, add margin, and that is your number; the default 10 is a starting point, not a decision. Enforce it in one place. The flag is per invocation, so a single manual upgrade without it silently trims the release back to 10 — at exactly the moment somebody is deploying by hand during an incident and might want a deep rollback. Put it in the deploy template, the wrapper script and the break-glass runbook, not in each pipeline's environment block. And resist using retention as a storage-pressure fix for the wrong problem: it bounds the number of records, never the size of one. A record too large to write is a chart problem or a release-splitting problem. ### Recovery Write down what happens when a namespace's records are lost — deleted by a cleanup job, dropped by a partial restore, or stranded by a backend switch. The workloads keep running; Helm simply forgets them. Recovery means re-establishing a release over live objects that still carry the previous release's ownership metadata, which is a deliberate adoption step rather than a re-install, and it goes far more smoothly when the chart version and values are recoverable from Git than when they were typed into a terminal. So the concrete asks are: chart and values in version control for every production release; deploys through a path that records what it passed; backups that include the release records or a documented acceptance that they are re-creatable; and a rehearsed procedure for adopting live objects back into a new release record. Rehearsed matters — the first time anyone tries this should not be during the incident that caused it. ### Governance of reads One more thing a lead owns: whatever values were supplied are stored in the record. A password passed to a chart is readable by anyone who can read Secrets in that namespace, whether or not the chart rendered it anywhere. In a multi-tenant cluster that makes namespace read access to release records a real privilege boundary, and it argues for charts that reference externally-managed credentials rather than accepting them as values — a chart-design decision driven entirely by how release state is stored. ### What I would land on Default backend; retention set from detection windows and enforced centrally; chart and values in Git so records are genuinely derived state; explicit read governance on namespaces where credentials pass through values; and `sql` reserved as a targeted answer to a measured problem, never as a fleet-wide default.