skip to content

Why is helm upgrade described as a one-shot apply rather than a reconcile loop?

level: juniorimportance: must knowfreq 72%

answer

  1. it has a beginning and an end
  2. what runs after the command exits
  3. drift waits for the next invocation
  4. revision recorded, process gone

basics

~20 s

helm upgrade runs once: it renders the chart, sends the result to the API server, records a new release revision and exits. Nothing then watches the cluster, so drift survives until somebody runs Helm again.

solid answer

~50 s

`helm upgrade` is a command with a beginning and an end. It merges values, renders the chart, applies the rendered objects, stores a new revision as a Secret named `sh.helm.release.v1.<name>.v<rev>`, and exits. From that moment Helm has no presence in the cluster: there is no Helm pod, no watch and no interval. If somebody edits a Deployment by hand, or a node dies and the workload comes back wrong, Helm neither notices nor reacts — the next `helm upgrade` or `helm rollback` is the only thing that re-asserts the chart. A controller is the opposite shape: it runs inside the cluster, watches the objects it owns and re-runs its own logic on every change and on a periodic resync. A command that ends versus a process that keeps running is exactly what people mean by one-shot apply versus reconciliation.

code

bash · 3 lines
bash
helm upgrade ledger ./ledger-umbrella --wait
kubectl scale deploy/ledger-api --replicas=7   # Helm is not running; nothing reverts this
helm upgrade ledger ./ledger-umbrella          # only now is the chart's value re-asserted

go deeper

for a junior

Be ready to say plainly that Helm is a command that runs, changes the cluster and exits, with nothing left behind except the release record. Know that drift stays until you run Helm again.

for a middle

Explain the actual steps of an upgrade — merge values, render, apply, write a new revision — and that Helm does not wait for workloads unless asked. Contrast that with a controller that watches and re-runs on events.

for a senior

Show where this bites in production: manual edits surviving, an upgrade reported green while pods never became ready, and the reflex to place backup or failover in a chart because that is where the config lives.

for a principal

Own the framing that Helm's unit of work is a recorded revision and a controller's is a continuous reconcile, and use it to decide which platform responsibilities can live in charts at all.

### What `helm upgrade` actually does, start to finish A single invocation does a fixed amount of work and then the process exits: 1. Load the chart (a directory or a `.tgz`) and merge values — chart `values.yaml` first, then each `-f` file, then `--set` overrides. 2. Render `templates/` with Go text/template to produce one manifest of Kubernetes objects. 3. Read the current release record — the Secret holding the latest revision — to learn what the previous revision put in the cluster and with which values. 4. Run the pre-upgrade hooks, send the rendered objects to the API server, run the post-upgrade hooks. 5. Write a new release record containing the new rendered manifest and the merged values, mark it deployed, and prune history beyond `--history-max` (default 10). 6. Exit with a status code. The release record is worth dwelling on, because it is Helm's entire memory. It is a Secret named `sh.helm.release.v1.<name>.v<rev>` in the release namespace, and it holds the full rendered manifest plus the values used. For a large umbrella chart — say one bundling an API subchart and a worker subchart for a payments ledger — that Secret can easily reach 1.1 MiB. It is read only when a `helm` command runs. No component in the cluster consumes it, reconciles from it, or alerts on it. ### What happens after the command exits: nothing This is the whole point of the comparison. Helm is a client-side package manager. Once `helm upgrade` returns there is no Helm process anywhere. Three consequences follow directly: * **Drift persists.** If somebody scales the ledger API Deployment by hand, or another tool rewrites an annotation, the cluster stays that way. The chart is not "the truth" in any enforced sense — it is the truth as of the last time somebody ran Helm. * **Failure is not reacted to.** If a pod crashes or a primary is lost at 03:00, Helm does not run. Whatever recovers the workload is something else: the built-in Kubernetes controllers for restarts and rescheduling, or a purpose-written controller for anything more clever. * **Helm does not even wait by default.** `helm upgrade` returns as soon as the API server has accepted the objects unless you ask it to wait. In Helm 4 `--wait` is strategy-valued — `watcher`, `legacy` or `hookOnly` — and omitting the flag means `hookOnly`, so the command finishes without waiting for workloads to become ready. Success from Helm means "accepted", not "running". ### A controller is the other shape A controller runs as a Deployment inside the cluster. It watches the objects it cares about, and on every change — plus on a periodic resync — it compares desired against actual and takes a step toward closing the gap. That is a loop with no end, driven by cluster events rather than by a human at a terminal. That difference is what decides where day-two work lives. Taking a backup on a schedule, promoting a replica when a primary is lost, walking a stateful workload through a major version migration one member at a time — none of these can be expressed as "a file that somebody applies", because they are triggered by things that happen in the cluster at times nobody chose. They need something that is already running when the event occurs. Installing a chart is triggered by a person or a pipeline; reconciliation is triggered by reality. ### The two are not rivals in practice The usual production shape is both. Helm packages and versions the operator itself — its controller Deployment, its service account and RBAC, its configuration — and most teams acquire an operator by installing somebody else's chart. Helm also renders the custom resources that ask the operator for a payments ledger of a given size and version. The chart therefore owns the *desired shape*: which version, how many instances, what configuration. The controller owns *what to do when reality diverges from that shape*. ### "But we run Helm from CI on every merge" Running `helm upgrade` from a pipeline on merge, or having an in-cluster delivery agent apply the chart repeatedly, does add a loop — but the loop belongs to that agent, not to Helm. Helm is still the one-shot step inside it. And that loop reacts to commits, not to the ledger losing its primary at 03:00; the two triggers are genuinely different, and a candidate who conflates them will place backup and failover in the wrong place. ### How to say it in an interview Helm's unit of work is a *revision*: a discrete, recorded, rollback-able change made when a human or pipeline asks for one. A controller's unit of work is a *reconcile*: a continuous, event-driven correction that nobody asks for. Everything else — who owns backups, why `helm rollback` cannot undo a data migration, why an operator still ships as a chart — follows from that one sentence.

  • If Helm does not run continuously, what does correct a crashed pod after a chart-installed Deployment is live?
    The built-in Kubernetes controllers do. The Deployment controller keeps the desired replica count, and the kubelet restarts containers. Helm handed those objects to the cluster and left; the cluster's own control loops keep them in shape. What no built-in controller does is anything application-specific — taking a backup, promoting a replica, migrating data — which is exactly the gap an operator fills.
  • Does a successful `helm upgrade` mean the new version is serving traffic?
    No. By default the command returns once the API server has accepted the objects. In Helm 4, omitting `--wait` means `hookOnly`, so it does not wait for workloads at all; `--wait=watcher` makes it wait on readiness. Even then, Helm can only judge what the objects report — for a custom resource handed to an operator, acceptance says nothing about whether the operator has finished its work.
  • Why can Helm still roll back if it is not running in the cluster?
    Because every revision's rendered manifest and values are stored in the release record Secret. `helm rollback` reads the target revision, applies that manifest, and writes it as a new revision — it never rewinds history. It is another one-shot command; it restores a manifest, not the state of anything that manifest asked another system to do.

Installing a package sets a machine up once and the installer exits; a service manager is the thing that keeps watching afterwards. Helm is the installer, an operator the service manager.

saying these in an interview costs you the question

  • Says Helm keeps the cluster matching the chart continuously
  • Thinks a Helm agent or controller runs in the cluster
  • Believes helm upgrade reverts manual edits as they happen
  • Assumes a successful upgrade means pods are ready
  • Claims Helm handles failover because it manages the release
  • Confuses the stored release record with a live reconciler

context

resolving links…