skip to content

Drift & Change Preview

Helm records the manifest it last applied, not the live object, so a hand-edited resource is invisible to helm status. Asked because seniors preview an upgrade before running it in production.

part ofHelmoverview, primer and where to startread it →
on this pageshow

questions

4

Why does `helm status` still report a release as deployed after someone hand-edits its live objects?

level: middleimportance: must knowfreq 62%

answer

  1. Helm is not a controller
  2. the record, not the cluster
  3. status reads the release Secret
  4. deployed means the last operation succeeded
  5. diff the stored manifest yourself

basics

~20 s

Helm stores the manifest it applied in the release record and never watches the cluster afterwards. helm status reads that record, so it reports the outcome of the last operation, not whether the live objects still match it.

solid answer

~50 s

Helm is a render-and-apply tool, not a controller. Install and upgrade write a release record — a Secret named `sh.helm.release.v1.<name>.v<rev>` — holding the rendered manifest, the merged values and a status, and `helm status`, `helm list` and `helm get manifest` all read only that record. `deployed` means the last operation finished successfully; it says nothing about the cluster since. There is no watch, no informer and no periodic reconcile, so a `kubectl edit`, a scaled replica count or a deleted object is invisible. Drift detection is therefore something you do explicitly: `helm get manifest reranker-api -n reco | kubectl diff -f -` dry-runs the stored manifest against the API server and prints the field-level differences against what is live. Whether the next upgrade reverts the edit is a separate question, decided by the apply path rather than by anything status reports.

code

bash · 3 lines
bash
# What Helm last applied, compared with what is actually running
helm get manifest reranker-api -n reco | kubectl diff -f -
echo "exit=$?"   # non-zero when the cluster no longer matches the record

go deeper

for a junior

Remember the one-liner: Helm records what it applied and never looks again, so status reflects the last command, not the cluster. Be ready to say which command you would run to compare the two.

for a middle

Explain the mechanism: the release record Secret, what it holds, and which commands read it rather than the API server. Then show the explicit comparison with helm get manifest piped into a server-side diff.

for a senior

Show operational judgement: how you would schedule a drift check, which differences are expected noise from other controllers, and how you decide between folding the change into the chart and re-running the upgrade to reassert it.

for a principal

Own the model choice. Explain when a client-side apply-once tool is the right delivery mechanism and when the team actually needs continuous reconciliation, and what the organisation gives up in audit and change control by relying on periodic manual diffs.

## What Helm actually stores Every successful `helm install`, `helm upgrade` or `helm rollback` finishes by writing a **release record**: by default a Secret in the release namespace named `sh.helm.release.v1.<release>.v<revision>`, containing a compressed blob with the chart metadata, the merged values, the rendered manifest, a status such as `deployed` or `failed`, and timestamps. `HELM_DRIVER` can point that storage at ConfigMaps, memory or SQL instead, but the content is the same. `helm status`, `helm list`, `helm history`, `helm get values` and `helm get manifest` all read that record and nothing else about the workload. That single fact is the whole answer. `deployed` is a statement about an **operation that finished**, not about the cluster today. Between operations Helm runs nothing: no watch, no informer, no controller, no interval. Nothing in Helm notices a `kubectl edit`, a `kubectl scale`, a mutating webhook rewriting a pod template, or another controller taking a field over. Delete every Deployment a release owns and `helm status` still prints `deployed`, with the same revision, and `helm list` still lists the release. The contrast worth naming in an interview is with a reconciling system — an in-cluster agent that continuously compares a desired state to live state and can report or correct divergence. Helm has never been that, by design; it is a client-side package manager that applies once per command and then goes away. Which of the two models you want is a delivery decision, not something to be fixed by looking harder at `helm status`. ## Three things called "the manifest" Much of the confusion here is vocabulary. There are three artefacts, and the question is only interesting when they disagree: 1. the YAML a **fresh render** produces from today's chart and values; 2. the manifest **stored in the release record**, which is what Helm last applied and what `helm get manifest` prints; 3. the **live objects** in the API server. They coincide for a moment after an upgrade and start diverging immediately: the API server defaults fields, controllers write status and take ownership of things like replica counts, and a human may edit something. `helm status` compares none of these; it just reads the status field inside (2)'s record. ## Doing the comparison yourself The standard move is to feed the stored manifest to a server-side diff: ```bash helm get manifest reranker-api -n reco | kubectl diff -f - ``` Each object is sent to the API server as a dry-run apply, so server defaulting is applied to your input before comparison and you see the field differences that a real apply would produce. It exits non-zero when there are differences, which is enough for a scheduled job to alert on. Know its blind spots: hook-annotated manifests are not part of what `helm get manifest` prints (they live in `helm get hooks`), resources installed from `crds/` are never updated or deleted by Helm at all so a template diff says nothing about them, and any field owned by another controller shows as a difference even though nobody drifted anything. ## Why this was never a status feature To report drift, Helm would have to fetch every object in the release, diff it field by field, and then decide which differences are *meaningful* — exactly the hard part that reconcilers spend their design budget on. Helm 3 had a status flag that listed the release's resources, and even that only printed names and readiness fetched with a get; it never compared fields against the stored manifest. Helm 4 removed it. Nothing in either version turns `helm status` into a drift report. ## What happens to the edit next time A good answer closes the loop without over-promising. Whether the hand edit survives the next `helm upgrade` is decided by the apply path — Helm 4 writes with server-side apply by default on install, while on upgrade the mode is inherited from how the release was previously applied — and by whether the chart even sets that field. If the chart does not template `spec.replicas`, an upgrade will typically leave a hand-scaled value alone; if it does, the upgrade asserts it back. "The next upgrade will fix it" is a guess, not a mechanism. The safe operational answer is: detect the difference deliberately with a diff, decide whether the cluster or the chart is right, and then either re-run the upgrade so the record and the cluster agree again, or fold the change into the chart.

  • Does a release listed as deployed by `helm list` prove the workload is healthy right now?
    No. `deployed` records that the last install or upgrade completed without error. Pods can crash-loop, be evicted or be deleted minutes later and the status never changes. Even a `--wait` strategy only gates the operation itself: it holds the command until the resources it applied became ready, then writes the record and stops caring. Health right now is a cluster question, answered by looking at the objects.
  • If Helm never watches the cluster, when does it look at live objects at all?
    Only while a command is running. Install, upgrade, rollback, uninstall and `helm test` talk to the API server for the duration of the operation, and a wait strategy keeps that conversation open until resources settle or the timeout fires. Read commands such as `helm get manifest` do not consult the cluster; they decode the stored release record. Between operations Helm holds no connection at all.
  • Someone deleted a Deployment that a release owns. What does Helm show, and how do you get it back?
    Nothing changes in Helm's output: the release is still `deployed` at the same revision, and `helm get manifest` still prints the Deployment because that is the stored text. Re-running `helm upgrade` with the same chart and values recreates it, since the apply covers every object in the manifest. `helm rollback` to the current revision does the same and costs you a new revision number.

The release record is a receipt for a delivery, not a camera pointed at the shelf. It tells you what was handed over and that the handover succeeded; it cannot tell you that someone has since taken a box off the shelf.

saying these in an interview costs you the question

  • Says helm status compares the chart against the running cluster
  • Thinks Helm runs a reconcile loop between upgrades
  • Treats status deployed as proof the pods are healthy
  • Believes helm get manifest reads the live objects
  • Assumes any hand edit is automatically reverted on the next upgrade
  • Confuses the release record Secret with a Secret the chart renders

context

open as a page

What does `helm diff upgrade` compare, and how is that different from `helm upgrade --dry-run=server`?

level: middleimportance: should knowfreq 48%

basics

~20 s

The helm-diff plugin's upgrade subcommand renders the new chart and prints a unified diff against the manifest stored in the release record. A server-side dry-run renders and sends the objects to the API server for validation and admission, printing the manifest rather than a diff.

open as a page

Piping `helm get manifest` into `kubectl diff` shows a large diff on a release nobody touched — why?

level: seniorimportance: should knowfreq 40%

basics

~20 s

Most of that diff is expected: fields other controllers own, server-assigned bookkeeping metadata, and objects the stored manifest never covered. Real drift is what a human or an unexpected actor wrote, so triage the diff by asking who owns each changed field.

open as a page

Before a production `helm upgrade`, what must a change preview prove, and where does it stop helping?

level: principalimportance: nice to knowfreq 30%

basics

~20 s

A preview should prove three separate things: the delta against what Helm last applied, that the cluster would accept the objects, and that nothing drifted since the last upgrade. It cannot prove the rollout will be healthy, and it never prevents the next out-of-band edit.

open as a page