skip to content

A team annotates a Kubernetes Deployment expecting its game-session pods to restart, but nothing rolls out. Why, and which annotations actually drive a Deployment's rollouts and revision history?

level: seniorimportance: should knowfreq 46%

answer

  1. which metadata block did you edit
  2. only the template hash matters
  3. restartedAt lives in the template
  4. revision renumbered on undo
  5. copied to the newest ReplicaSet

basics

~20 s

Only a change to the pod template starts a rollout, so annotations on the Deployment's own metadata never restart pods. Annotations under spec.template.metadata do, which is how kubectl rollout restart works, while deployment.kubernetes.io/revision numbers each rollout.

solid answer

~40 s

The Deployment controller hashes `spec.template` into the `pod-template-hash` label, and a rollout — a new ReplicaSet — happens only when that template changes. `kubectl annotate deployment game-session …` edits `metadata.annotations`: the API server bumps `metadata.generation`, the controller copies the key onto the newest ReplicaSet, and the running pods are untouched. To roll pods, change `spec.template.metadata.annotations`: `kubectl rollout restart` writes a `kubectl.kubernetes.io/restartedAt` timestamp there, and a checksum-of-config annotation there rolls pods when a ConfigMap changes. For history, the controller stamps `deployment.kubernetes.io/revision` on each ReplicaSet and mirrors the current value onto the Deployment. `kubectl rollout undo` promotes an old ReplicaSet under a new, higher revision and records its old number in `deployment.kubernetes.io/revision-history`. `kubectl rollout history` shows each revision's `kubernetes.io/change-cause`.

code

bash · 5 lines
bash
kubectl annotate deployment game-session games.example.com/reloaded=2026-09-17
kubectl rollout restart deployment/game-session
kubectl get deployment game-session -o jsonpath='{.spec.template.metadata.annotations}{"\n"}'
kubectl get rs -l app.kubernetes.io/name=game-session -o 'custom-columns=NAME:.metadata.name,REVISION:.metadata.annotations.deployment\.kubernetes\.io/revision'
kubectl rollout history deployment/game-session

go deeper

for a junior

Recall that only pod-template changes start a Deployment rollout, and that kubectl rollout restart is the supported way to restart its pods.

for a middle

Explain the pod-template-hash mechanism, the difference between the two annotation maps, and how the revision annotation numbers each ReplicaSet.

for a senior

Diagnose the no-rollout incident from the YAML: which block was edited, whether observedGeneration caught up, whether the Deployment is paused, and how undo renumbers revisions.

for a principal

Decide how config changes should roll pods across the platform: checksum annotations rendered by tooling, restart-on-change conventions, or immutable versioned ConfigMaps, weighing noise against drift.

## The incident A **multiplayer game-session backend** runs as a Deployment on a **5-node edge cluster in a retail store**. The team rotates a TLS bundle in a mounted Secret and, wanting fresh pods, runs `kubectl annotate deployment game-session games.example.com/reloaded=2026-09-17`. The command succeeds, `metadata.generation` goes up, and not a single pod restarts. The question is really about **where on the object an annotation lives**. ## Two annotation maps on one Deployment A Deployment carries two separate metadata blocks: | Location | Belongs to | Effect of editing it | |---|---|---| | `metadata.annotations` | The Deployment object | Copied to the newest ReplicaSet; pods unchanged | | `spec.template.metadata.annotations` | Every pod the template creates | Changes the template, so a rollout starts | The Deployment controller computes a hash of `spec.template` and puts it into the `pod-template-hash` label on the ReplicaSet and its pods. A **rollout** means the controller finds no ReplicaSet matching the current template hash, creates one, and shifts replicas to it under the rolling-update rules. Anything outside the template — the Deployment's own labels and annotations — cannot change that hash. ## What editing the Deployment's own annotations does The API server's update logic for Deployments bumps `metadata.generation` when either the spec **or** `metadata.annotations` change. The reason is the copy step: 1. The controller sees the new generation and syncs. 2. It copies the Deployment's annotations onto the **newest ReplicaSet**, skipping its own bookkeeping keys: `kubectl.kubernetes.io/last-applied-configuration` and the `deployment.kubernetes.io/` revision and replica keys. 3. It records the generation in `status.observedGeneration`. 4. Pods are never touched, because the template hash is unchanged. That copy is what makes `kubernetes.io/change-cause` work: set on the Deployment, it lands on the ReplicaSet, and `kubectl rollout history` prints it per revision. ## How to actually roll the pods Put the annotation **inside the template**: - `kubectl rollout restart deployment/game-session` sets `kubectl.kubernetes.io/restartedAt` to the current time under `spec.template.metadata.annotations`. It refuses a paused Deployment. - A **config checksum** annotation in the template, written by whatever renders the manifest, holds a hash of the ConfigMap or Secret content. When the content changes, the hash changes, the template changes, and a normal rolling update follows. - Editing any real template field works too — raising the CPU request to `350m` is itself a template change. Every template annotation also appears on the new pods, so choose keys that are harmless there. | Approach | Rolls pods when | Leaves a record | |---|---|---| | `kubectl rollout restart` | Someone runs it by hand or from a script | A timestamp in the template, so a new revision | | Config checksum in the template | The rendered config content changes | A new revision per distinct config | | Editing a real template field | The field value changes | A new revision with the changed spec | The restart command is right for a one-off such as the TLS rotation above; a checksum suits config that changes through the normal delivery path, because the rollout is then tied to the content rather than to someone remembering to restart. Both create an ordinary revision, so both can be undone like any other rollout. ## Revision annotations and rollback The controller keeps rollout history in annotations, not in a separate object: - `deployment.kubernetes.io/revision` on each ReplicaSet: the newest gets the highest number, which is mirrored onto the Deployment. - `deployment.kubernetes.io/revision-history` on a ReplicaSet that has been reused: a comma-separated list of the numbers it held before. - `deployment.kubernetes.io/desired-replicas` and `deployment.kubernetes.io/max-replicas` on ReplicaSets, which proportional scaling uses. Suppose ReplicaSets hold revisions 3, 4 and 5 and you run `kubectl rollout undo deployment/game-session --to-revision=4`. The Deployment's template is set back to revision 4's template; its hash matches the existing ReplicaSet, so that ReplicaSet is scaled up and renumbered to **6** (the maximum plus one), with `4` recorded in `revision-history`. Revision 4 then no longer appears in `kubectl rollout history`. How many old ReplicaSets remain to roll back to is set by `spec.revisionHistoryLimit`, which defaults to 10. ## What to check when nothing rolls - Compare `.metadata.annotations` with `.spec.template.metadata.annotations` in `kubectl get deployment -o yaml`. - Confirm `status.observedGeneration` equals `metadata.generation`, so you know the controller saw the edit. - Check `spec.paused`: a paused Deployment does not roll even when the template changes. - Inspect ReplicaSets with `kubectl get rs -l app.kubernetes.io/name=game-session` and read their revision annotations.

  • Why does `kubectl rollout restart` fail on some Deployments?
    It refuses a paused Deployment with an error telling you to run `kubectl rollout resume` first. A paused Deployment does not act on template changes, so stamping `restartedAt` would do nothing until it is resumed; kubectl makes that explicit instead of reporting a restart that never happens.
  • Which Deployment annotations does the controller deliberately not copy to its ReplicaSets, and why?
    It skips `kubectl.kubernetes.io/last-applied-configuration` and its own bookkeeping keys: `deployment.kubernetes.io/revision`, `revision-history`, `desired-replicas` and `max-replicas`. The applied manifest belongs to the Deployment only, and the revision and replica values are set per ReplicaSet by the controller itself, so copying the Deployment's copies would overwrite correct values with stale ones.
  • After `kubectl rollout undo`, why can you no longer find the revision you rolled back to in the history?
    Rollback reuses the existing ReplicaSet whose template matches and gives it the next revision number, so revision 4 becomes revision 6. The old number survives only in that ReplicaSet's `deployment.kubernetes.io/revision-history` annotation, which `kubectl rollout history` does not list as its own row.

Writing a note on the cover of a recipe binder does not change the dish; the kitchen only cooks something new when a page inside, the recipe itself, is edited.

saying these in an interview costs you the question

  • Any annotation change on a Deployment triggers a rolling update.
  • Annotations set on the Deployment are copied onto every running pod.
  • kubectl rollout undo restores the old revision number unchanged.
  • kubectl rollout restart deletes all pods at once to restart them.
  • Editing Deployment annotations leaves metadata.generation untouched.