skip to content

Why can you run helm rollback on a release but not undo a kubectl apply -k the same way?

level: middleimportance: must knowfreq 62%

answer

  1. One side keeps a record in the cluster
  2. A Secret per revision in the namespace
  3. Rolling back moves forward, not back
  4. Old revisions do not live forever
  5. Manifests come back; data does not

basics

~20 s

Each helm install or upgrade stores the rendered manifest, its values and a status as a release revision in a namespace Secret, so helm rollback can re-apply an earlier one. kubectl apply -k stores nothing; undoing it means rebuilding the older source.

solid answer

~50 s

Helm writes a record for every release revision — a Secret named `sh.helm.release.v1.<name>.v<rev>` in the release namespace holding the rendered manifest, the values that produced it, the chart metadata and a status. `helm history` lists them and `helm rollback <release> <rev>` re-applies that stored manifest, writing a **new** revision rather than rewinding: a release at revision 34 rolled back to 30 becomes revision 35. Two limits bite in practice. `--history-max` defaults to 10, so on a long-lived release only the newest revisions still exist — an early revision is simply gone. And rollback restores manifests, not data or objects Helm never created, such as the PersistentVolumeClaims a StatefulSet's `volumeClaimTemplates` makes. With `kubectl apply -k` there is no record at all: recovery means checking out the older source, rebuilding it and applying again — which also will not remove resources the newer version added.

code

bash · 9 lines
bash
# What Helm knows about a long-lived release
helm history redis-cache -n payments
kubectl get secret -n payments -l owner=helm,name=redis-cache

# Re-apply revision 30's stored manifest; this creates revision 35
helm rollback redis-cache 30 -n payments

# Keep more than the default 10 revisions on the next upgrade
helm upgrade redis-cache ./charts/redis -n payments --history-max 25

go deeper

for a junior

Know that Helm remembers each install and upgrade as a numbered revision inside the cluster, that helm history lists them, and that helm rollback re-applies an earlier one. A plain apply keeps no such list.

for a middle

Explain the mechanism: a Secret per revision holding the rendered manifest, values and status; upgrades diffing against the stored previous manifest; rollback writing a new revision rather than rewinding; --history-max defaulting to 10.

for a senior

Demonstrate incident judgement: check whether the target revision still exists before promising a rollback, know that data and controller-created volumes do not come back, and be able to say when reverting the source and re-applying is the safer move.

for a principal

Own the tradeoff between a stateful delivery tool with an in-cluster history and a stateless apply whose only truth is the source tree, including where recovery procedures, etcd growth and the second source of truth land your organisation.

### What Helm stores, and where Every `helm install`, `helm upgrade` and `helm rollback` writes a **release revision** into the cluster. By default the storage driver is `secret`, and the object is a Secret in the release's namespace named `sh.helm.release.v1.<release>.v<revision>`. Inside it, base64'd and gzipped, is the whole picture of that revision: the fully rendered manifest, the coalesced values, the chart metadata, a status (`deployed`, `superseded`, `failed`, `pending-upgrade` and so on) and timestamps. `HELM_DRIVER` can move that storage to `configmap`, `memory` or `sql`, but the shape of the record does not change. That record, not the CLI, is what makes Helm stateful. `helm list` finds releases by looking for those Secrets. `helm history <release>` prints one line per revision. `helm get manifest <release>` decodes the manifest of the current revision, and `helm get values` the values. An upgrade compares the new render against the **stored previous manifest**, which is how Helm knows that a resource you deleted from the chart should be deleted from the cluster — a plain apply of a smaller manifest set leaves the old objects running. ### What rollback actually does Take a Redis chart with a StatefulSet and a PersistentVolumeClaim whose release has been upgraded steadily for two years and now sits at revision 34. `helm rollback redis-cache 30` does not rewind history. It reads revision 30's stored manifest, applies it, marks the current revision superseded, and writes the result as **revision 35** whose content equals revision 30's. `helm history` afterwards shows 35 lines' worth of story, the newest annotated as a rollback. That forward-only design is deliberate: the audit trail is append-only, and rolling back a rollback is just another rollback. Two caveats decide whether the command helps you in an incident. **History is pruned.** `--history-max` defaults to 10, and it is applied on upgrade: older revision Secrets are deleted once the count exceeds the maximum. On a 34-revision release with the default, revisions 1 to 24 no longer exist, so `helm rollback redis-cache 3` fails — there is nothing to read. Teams that want deep history raise `--history-max` knowingly, accepting that each revision Secret carries a full copy of the rendered manifest and therefore costs space in etcd. **Rollback restores manifests, not the world.** It re-applies YAML. It does not restore data, and it does not touch objects Helm never created. The PersistentVolumeClaims that a StatefulSet's `volumeClaimTemplates` generates are made by the StatefulSet controller, not by Helm, so they appear in no release manifest: rolling the chart back to a revision with a smaller `storage` request changes the template, and the existing claims stay exactly as they are. Anything annotated `helm.sh/resource-policy: keep` is likewise outside the diff. And a rollback whose old revision referenced an image tag that has since been deleted from the registry produces pods that will not start. ### The other side: a stateless apply `kubectl apply -k` builds manifests and applies them. Nothing records that this happened beyond the objects themselves and the `kubectl.kubernetes.io/last-applied-configuration` annotation on each one, which describes only the last apply, not a series. There is no command that answers "what did we deploy here, and what did we deploy before that?" — those answers live in the Git history of the source and in your CI logs, not in the cluster. So undoing it is a source-control operation: check out the commit that built the previous variant, rebuild it, apply again. That works, and for many teams it is entirely sufficient — arguably it is more honest, since the source is the only truth and no cluster-side record can drift away from it. But notice what it demands. The older source must still build the same output today, which fails if it referenced a floating base, a tag that moved, or a generator whose version changed. Re-applying also does not remove anything: a resource introduced by the newer version is not in the older build, and a plain apply has no way to know it should go. Deleting it is a separate, manual step — or a job for a pruning mechanism you configure deliberately, with its own blast radius. ### How to answer Say that the difference is not `rollback` versus no `rollback`, it is stateful versus stateless delivery. Helm buys an in-cluster, queryable history and a one-command reversal, at the price of a second source of truth that can disagree with Git, revision Secrets in etcd, and a pruning window. A plain apply buys simplicity and exactly one source of truth, at the price of doing recovery through the source tree and having no in-cluster answer to "what is running here". Then show the edges: rollback creates a new revision, history is capped at 10 by default, and neither approach restores data.

  • You delete a resource from a chart and upgrade. What happens to the live object, and why is that different from a plain apply?
    Helm diffs the new render against the manifest stored with the previous revision, sees the resource is gone, and deletes it — unless it carries `helm.sh/resource-policy: keep`. A plain apply of a smaller manifest set has nothing to diff against, so the old object simply keeps running until something prunes it deliberately.
  • Why might helm rollback succeed and still leave the release broken?
    Because it only re-applies YAML. Data written by the newer version is still there, a database migration is not reversed, PersistentVolumeClaims created by a StatefulSet controller are untouched, and an image tag referenced by the old revision may no longer exist in the registry. Rollback restores intent, not the state of the world.
  • What happens if the release Secrets are deleted from the namespace?
    Helm loses the release entirely: `helm list` no longer shows it and `helm upgrade` treats it as absent, while the objects keep running with nothing tracking them. Adopting them back means recreating the ownership metadata or installing over them with Helm taking ownership — which is why those Secrets deserve the same care as any other cluster state.

saying these in an interview costs you the question

  • Says helm rollback rewinds history and deletes later revisions
  • Believes rollback restores application data or volumes
  • Thinks all revisions are kept forever by default
  • Claims the release history lives on the engineer's laptop
  • Says re-applying older source removes resources the newer one added

context

resolving links…