Your service reads its configuration file only at process startup. How do you make a Kubernetes Deployment actually roll out when its ConfigMap changes, and how do you keep that change revertible?
answer
- Deployment only watches the Pod template
- checksum/config annotation perturbs the template hash
- Kustomize configMapGenerator = name hash + reference rewrite
- rollout undo restores template, not a mutated ConfigMap
- kubectl rollout restart = emergency lever
basics
~20 sMake the config change alter the Pod template: put a hash of the ConfigMap content in a Pod-template annotation, or give the ConfigMap a content-hash suffix in its name so the reference changes. Either turns the edit into a normal rolling update you can roll back.
solid answer
~60 sKubernetes never restarts Pods because a referenced ConfigMap changed, so you have to make the change visible in the Pod template — that is the only thing a Deployment watches. Two standard techniques: - **Checksum annotation.** Add `annotations: {checksum/config: <sha256 of the ConfigMap>}` to the Pod template. Helm charts do this with `sha256sum` over the rendered ConfigMap. Any content change alters the template hash, so the Deployment performs a rolling update with the usual readiness gating. - **Hashed ConfigMap name.** Kustomize's `configMapGenerator` appends a content hash to the object name and rewrites every reference. Now each config version is a distinct, immutable object, and `kubectl rollout undo` restores both the old template *and* its old ConfigMap. The second is strictly better for rollback: with a mutated shared ConfigMap, undoing the Deployment restores the old Pod template but the Pods still read today's config. The cost is orphaned old ConfigMaps to garbage-collect. For emergencies `kubectl rollout restart deployment/app` recreates Pods without editing anything. `Reloader`-style controllers automate the patch but move the trigger outside your manifests.
code
yaml · 33 lines# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deployment.yaml
configMapGenerator:
- name: app-config
files:
- application.yaml
generatorOptions:
disableNameSuffixHash: false # keep the content hash
---
# deployment.yaml references the plain name; kustomize rewrites it to app-config-<hash>
apiVersion: apps/v1
kind: Deployment
metadata:
name: app
spec:
replicas: 3
selector:
matchLabels: {app: app}
template:
metadata:
labels: {app: app}
spec:
containers:
- name: app
image: example/app:1.0
volumeMounts:
- {name: config, mountPath: /etc/app, readOnly: true}
volumes:
- name: config
configMap: {name: app-config}go deeper
Know that editing a ConfigMap alone does nothing and that kubectl rollout restart recreates the Pods.
Explain that the Pod template hash is the rollout trigger and show the checksum-annotation pattern.
Compare checksum annotations with hashed ConfigMap names on rollback fidelity, discuss garbage collection of old objects, and describe the failure modes when a new config is bad.
Define the config-delivery contract for the org: config versioned and released like images, one change path, retention aligned with revision history, and an explicit stance on hot reload versus progressive rollout.
## The mechanism you are working with A Deployment owns ReplicaSets; a ReplicaSet is created for each distinct Pod template. The Deployment controller compares the desired Pod template hash to the existing ReplicaSets and, if it differs, creates a new ReplicaSet and scales it up while scaling the old one down under `maxSurge`/`maxUnavailable`. That is the *entire* trigger. A ConfigMap referenced by the template is not part of the template's hash, so mutating it is invisible to the controller — no new ReplicaSet, no rollout, no revision recorded. Hence the design rule: **if you want a rollout, change the Pod template.** ## Technique 1 — checksum annotation Add an annotation to `spec.template.metadata.annotations` whose value is a hash of the configuration content. In Helm: `checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}`. The annotation is meaningless to Kubernetes — it exists purely to perturb the template hash. Any change to the rendered ConfigMap changes the annotation, which changes the template, which creates a new ReplicaSet and a normal rolling update with readiness probes, surge control and a recorded revision. Strengths: minimal machinery, works with one ConfigMap object name that never changes, easy to reason about. Weakness: the ConfigMap object is still *mutated in place*. `kubectl rollout undo` restores the previous Pod template (including the old checksum annotation), but the ConfigMap now on the cluster contains the new content, so the recreated Pods read the bad config while the annotation claims otherwise. Rollback is incomplete — you must also revert the ConfigMap, usually by re-running the previous release. ## Technique 2 — hashed ConfigMap names Generate the ConfigMap with a content hash in its name — `app-config-7c9f2h4kd8` — and rewrite the workload's reference to match. Kustomize's `configMapGenerator` does this automatically; Helm users can emulate it by templating the name with a `sha256sum` suffix. Now each version of the configuration is a separate immutable object. A config change alters the *name* in the Pod template, which triggers the rollout; `kubectl rollout undo` restores the previous template, whose reference points at the previous, still-present ConfigMap, so the rollback is complete and truthful. This composes beautifully with `immutable: true`: the object can never be edited in place, which also lets kubelets stop watching it. The cost is lifecycle management. Old ConfigMaps are not garbage-collected by default (nothing owns them), so you accumulate objects. Options: let Kustomize/Argo CD pruning remove unreferenced ones, set `ownerReferences` to a ReplicaSet, or run a periodic sweep — but never prune so aggressively that the object a rollback needs has already vanished. Keep at least as many generations as `revisionHistoryLimit`. ## Technique 3 — force a restart `kubectl rollout restart deployment/app` patches the Pod template with a `kubectl.kubernetes.io/restartedAt` timestamp annotation. It is a real rolling restart (not a delete-all), respects surge settings and probes, and needs no manifest changes. It is the right tool for an incident or a one-off, and the wrong tool as a routine deployment step, because the cluster state no longer tells you which config version is running. ## Technique 4 — reload controllers and in-app watching Controllers such as Reloader watch ConfigMaps and patch annotated Deployments for you; the trigger then lives in the cluster rather than in your pipeline, which some teams like and others consider a hidden actor. Alternatively the application itself watches the mounted directory and reloads (remembering that the kubelet swaps the `..data` symlink, so watch the directory, not the file). In-app reload is fastest but skips readiness probes and applies to every replica at once — a bad value has no canary and no automatic revert. Reserve it for components designed for it, such as proxies with validated config reloads. ## Adjacent failure modes to mention - **Missing key or ConfigMap:** the new Pods fail with `CreateContainerConfigError` (env) or `FailedMount` (volume). Because the rollout is gated by readiness, the old ReplicaSet keeps serving and `progressDeadlineSeconds` eventually marks the Deployment failed — which is exactly why routing config changes through a rollout is safer than live mutation. - **`optional: true`** turns a missing reference into a silently absent value; convenient, but it can start a Pod with defaults nobody intended. - **Shared ConfigMaps:** if several Deployments reference one object, an edit ripples to all of them, but only the ones you rolled will pick it up — producing a fleet where identical Pod specs run different configuration. Hashed names make this impossible. - **DaemonSets and StatefulSets** honour the same rule with their own update strategies (`OnDelete` StatefulSets will not roll at all until you delete Pods). ## How to answer Start with why nothing happens by default (the Pod template is the only trigger), give the checksum annotation as the common fix, then argue for hashed names on rollback grounds, and finish with the emergency lever (`rollout restart`) and the caveat about in-app hot reload bypassing progressive delivery.
- You mutate a shared ConfigMap in place and use a checksum annotation. Why is `kubectl rollout undo` not a true rollback?Undo restores the previous Pod template, including its old checksum annotation, but the ConfigMap object on the cluster still holds the new content. The recreated Pods therefore read the bad configuration while the template claims the old version. Hashed ConfigMap names avoid this because the old template references a different, still-existing object.
- What happens to the rollout if the new ConfigMap is missing a key the container references?New Pods cannot start — `CreateContainerConfigError` for env references, `FailedMount` for volumes — so they never become ready. The rolling update stalls with the old ReplicaSet still serving traffic, and after `progressDeadlineSeconds` the Deployment is marked as failed to progress. This gated failure is a strong argument for routing config changes through rollouts rather than live edits.
- When is in-application hot reload of a mounted config file the right choice?When the component is built for it and validates the new configuration before applying it — proxies and gateways such as Envoy or nginx are the usual cases — and when restart cost is high (large caches, long connection drain). Even then it is worth pairing with a rollout-based path for risky changes, because hot reload applies to every replica at once with no readiness gating and no automatic revert.
saying these in an interview costs you the question
- Believing Kubernetes restarts Pods automatically when a ConfigMap changes
- Deleting Pods manually as the standard config-deploy step instead of changing the template
- Assuming rollout undo also reverts a ConfigMap that was edited in place
- Pruning old hashed ConfigMaps so aggressively that rollback breaks
- Relying on live volume propagation for a service that only reads config at startup