On a three-node kubeadm cluster where every node runs the control plane, what is the sequence to upgrade Kubernetes by one minor version?

middleshould knowfreq 60%

answer

one node, one etcd loss
plan, apply, then node
kubelet binary is manual
addons wait for last control plane
backups under /etc/kubernetes/tmp

basics

~20 s

Upgrade kubeadm on the first node, run kubeadm upgrade plan and kubeadm upgrade apply, then upgrade its kubelet; on each other node upgrade kubeadm, run kubeadm upgrade node, then its kubelet, draining one node at a time.

solid answer

~40 s

Node by node. On the first control-plane node: point the package repository at the new minor, upgrade the kubeadm package, run kubeadm upgrade plan to see targets and blockers, then kubeadm upgrade apply v1.37.x. That rewrites the static-pod manifests for etcd, kube-apiserver, kube-controller-manager and kube-scheduler, backs up the old ones, renews component certificates and refreshes the kubelet configuration. Then drain the node, upgrade the kubelet and kubectl packages, run systemctl daemon-reload and restart the kubelet, and uncordon. On the second and third nodes run kubeadm upgrade node, which takes no version argument, then do the same kubelet steps. CoreDNS and kube-proxy are upgraded only once the last control-plane node is done. Take an etcd snapshot first, and never have two nodes down: three etcd members survive only one loss.

Node by node. On the first control-plane node: point the package repository at the new minor, upgrade the `kubeadm` package, run `kubeadm upgrade plan` to see targets and blockers, then `kubeadm upgrade apply v1.37.x`. That rewrites the static-pod manifests for etcd, `kube-apiserver`, `kube-controller-manager` and `kube-scheduler`, backs up the old ones, renews component certificates and refreshes the kubelet configuration. Then drain the node, upgrade the `kubelet` and `kubectl` packages, run `systemctl daemon-reload` and restart the kubelet, and uncordon. On the second and third nodes run `kubeadm upgrade node`, which takes no version argument, then do the same kubelet steps. CoreDNS and `kube-proxy` are upgraded only once the last control-plane node is done. Take an etcd snapshot first, and never have two nodes down: three etcd members survive only one loss.

code

bash · 15 lines

bash

# node 1 (first control-plane node)
sudo sed -i 's/v1.36/v1.37/' /etc/apt/sources.list.d/kubernetes.list
sudo apt-mark unhold kubeadm && sudo apt-get update
sudo apt-get install -y kubeadm='1.37.0-*' && sudo apt-mark hold kubeadm
sudo kubeadm upgrade plan
sudo kubeadm upgrade apply v1.37.0

# then drain node 1 and upgrade its kubelet
sudo apt-mark unhold kubelet kubectl
sudo apt-get install -y kubelet='1.37.0-*' kubectl='1.37.0-*'
sudo apt-mark hold kubelet kubectl
sudo systemctl daemon-reload && sudo systemctl restart kubelet

# nodes 2 and 3: after upgrading the kubeadm package
sudo kubeadm upgrade node

go deeper

for a junior

Know the order of commands: upgrade kubeadm, plan, apply on the first node, node on the others, and upgrade the kubelet by hand.

for a middle

Explain what apply rewrites (static-pod manifests, certificates, kubelet config) and what it leaves alone (the kubelet binary, addons until the last control plane).

for a senior

Protect the cluster while you do it: etcd snapshot first, one node at a time for quorum, capacity for two nodes, and health checks between nodes.

for a principal

Decide how much of this to automate and rehearse on a copy of the cluster, since a failed hop on three stacked nodes has little room for error.

The shape of the cluster

The example cluster runs a clinical-records portal on three machines. Each is a stacked control-plane node: it runs kube-apiserver, kube-controller-manager, kube-scheduler and an etcd member as static pods (Pods the kubelet starts from files in /etc/kubernetes/manifests), and it also runs portal Pods. Two facts shape the procedure:

etcd quorum with three members is two, so the cluster survives exactly one node being out. Upgrade strictly one node at a time.
Capacity: draining one node leaves two thirds of the capacity, so the portal must fit on two nodes during each node's window.

Before starting, take an etcd snapshot and read the release notes for the target minor.

Step 1: the first control-plane node

Switch the package repository. The community package repositories are split per minor version, so the repository definition must be changed to the new minor before the new packages are visible.
Upgrade kubeadm only, and check kubeadm version.
Run kubeadm upgrade plan. It checks the cluster, lists the versions you may upgrade to, shows Components that must be upgraded manually (the kubelets), and prints a table of kubeadm-managed configuration with a MANUAL UPGRADE REQUIRED column.
Run kubeadm upgrade apply v1.37.x. Its phases, in order: preflight, control-plane, upload-config, kubeconfig, kubelet-config, bootstrap-token, addon, post-upgrade.

What kubeadm upgrade apply actually does:

Rewrites static-pod manifests for etcd (unless --etcd-upgrade=false) and the three control-plane components, one at a time, waiting for each to come back and restoring the previous manifest if it does not.
Keeps backups under /etc/kubernetes/tmp, in directories named kubeadm-backup-manifests-<timestamp> and kubeadm-backup-etcd-<timestamp>.
Renews component certificates by default (--certificate-renewal defaults to true).
Updates the kubelet-config ConfigMap in kube-system and writes this node's kubelet configuration file.
Skips addons for now: CoreDNS and kube-proxy are upgraded only when every control-plane instance runs the new version, so on the first of three nodes it prints that it is skipping them.

Step 2: the kubelet on that node

kubeadm upgrade apply does not touch the kubelet binary.

Drain the node (cordon plus eviction of its Pods).
Upgrade the kubelet and kubectl packages.
systemctl daemon-reload, then systemctl restart kubelet.
Uncordon the node and confirm it is Ready at the new version.

Step 3: the other two nodes

For each remaining node, in turn:

Upgrade the kubeadm package.
Run kubeadm upgrade node. It takes no version argument: it reads the target from the cluster's stored configuration, which kubeadm upgrade apply already updated. On a control-plane node it upgrades the local static pods; on any node it refreshes the kubelet configuration.
Drain, upgrade kubelet and kubectl, reload and restart the kubelet, uncordon.

On the third node - the last control-plane instance - the addon phase finally upgrades CoreDNS and kube-proxy.

Commands compared

Command	Where it runs	What it changes
`kubeadm upgrade plan`	first control-plane node	nothing; reports targets and blockers
`kubeadm upgrade diff`	any control-plane node	nothing; shows manifest differences
`kubeadm upgrade apply <version>`	exactly one control-plane node per hop	cluster config, local static pods, certificates, kubelet config
`kubeadm upgrade node`	every other node, control plane or worker	local static pods if control plane, kubelet config

Worker-only nodes, if the cluster had them, would follow Step 3 after all control-plane nodes: kubeadm upgrade node only refreshes their kubelet configuration.

Checks between nodes

kubectl get nodes shows the new VERSION for the finished node and Ready everywhere.
Control-plane Pods in kube-system are running on the finished node.
The portal's Pods are back to their desired count before the next node is drained.
etcd reports three healthy members.
kubeadm upgrade plan on the next node, after its kubeadm package is upgraded, still reports the cluster version and flags anything unexpected.

If a node fails mid-way, stop: do not start the next node while one is unhealthy, because the remaining two are all that keep etcd quorate and the portal served.

## The shape of the cluster The example cluster runs a clinical-records portal on three machines. Each is a **stacked control-plane node**: it runs `kube-apiserver`, `kube-controller-manager`, `kube-scheduler` and an etcd member as **static pods** (Pods the kubelet starts from files in `/etc/kubernetes/manifests`), and it also runs portal Pods. Two facts shape the procedure: - **etcd quorum** with three members is two, so the cluster survives exactly one node being out. Upgrade strictly one node at a time. - **Capacity**: draining one node leaves two thirds of the capacity, so the portal must fit on two nodes during each node's window. Before starting, take an etcd snapshot and read the release notes for the target minor. ## Step 1: the first control-plane node 1. **Switch the package repository.** The community package repositories are split per minor version, so the repository definition must be changed to the new minor before the new packages are visible. 2. **Upgrade kubeadm only**, and check `kubeadm version`. 3. **Run `kubeadm upgrade plan`.** It checks the cluster, lists the versions you may upgrade to, shows `Components that must be upgraded manually` (the kubelets), and prints a table of kubeadm-managed configuration with a `MANUAL UPGRADE REQUIRED` column. 4. **Run `kubeadm upgrade apply v1.37.x`.** Its phases, in order: `preflight`, `control-plane`, `upload-config`, `kubeconfig`, `kubelet-config`, `bootstrap-token`, `addon`, `post-upgrade`. What `kubeadm upgrade apply` actually does: - **Rewrites static-pod manifests** for etcd (unless `--etcd-upgrade=false`) and the three control-plane components, one at a time, waiting for each to come back and restoring the previous manifest if it does not. - **Keeps backups** under `/etc/kubernetes/tmp`, in directories named `kubeadm-backup-manifests-<timestamp>` and `kubeadm-backup-etcd-<timestamp>`. - **Renews component certificates** by default (`--certificate-renewal` defaults to true). - **Updates the `kubelet-config` ConfigMap** in `kube-system` and writes this node's kubelet configuration file. - **Skips addons** for now: CoreDNS and `kube-proxy` are upgraded only when every control-plane instance runs the new version, so on the first of three nodes it prints that it is skipping them. ## Step 2: the kubelet on that node `kubeadm upgrade apply` does not touch the kubelet binary. 1. Drain the node (cordon plus eviction of its Pods). 2. Upgrade the `kubelet` and `kubectl` packages. 3. `systemctl daemon-reload`, then `systemctl restart kubelet`. 4. Uncordon the node and confirm it is `Ready` at the new version. ## Step 3: the other two nodes For each remaining node, in turn: 1. Upgrade the `kubeadm` package. 2. Run `kubeadm upgrade node`. It takes **no version argument**: it reads the target from the cluster's stored configuration, which `kubeadm upgrade apply` already updated. On a control-plane node it upgrades the local static pods; on any node it refreshes the kubelet configuration. 3. Drain, upgrade `kubelet` and `kubectl`, reload and restart the kubelet, uncordon. On the third node - the last control-plane instance - the `addon` phase finally upgrades CoreDNS and `kube-proxy`. ## Commands compared | Command | Where it runs | What it changes | |---|---|---| | `kubeadm upgrade plan` | first control-plane node | nothing; reports targets and blockers | | `kubeadm upgrade diff` | any control-plane node | nothing; shows manifest differences | | `kubeadm upgrade apply <version>` | exactly one control-plane node per hop | cluster config, local static pods, certificates, kubelet config | | `kubeadm upgrade node` | every other node, control plane or worker | local static pods if control plane, kubelet config | Worker-only nodes, if the cluster had them, would follow Step 3 after all control-plane nodes: `kubeadm upgrade node` only refreshes their kubelet configuration. ## Checks between nodes - `kubectl get nodes` shows the new `VERSION` for the finished node and `Ready` everywhere. - Control-plane Pods in `kube-system` are running on the finished node. - The portal's Pods are back to their desired count before the next node is drained. - etcd reports three healthy members. - `kubeadm upgrade plan` on the next node, after its kubeadm package is upgraded, still reports the cluster version and flags anything unexpected. If a node fails mid-way, stop: do not start the next node while one is unhealthy, because the remaining two are all that keep etcd quorate and the portal served.

What happens if a control-plane static pod does not come back healthy during kubeadm upgrade apply?
kubeadm waits for each rewritten component, and on failure restores the previous manifest from its backup directory (and the etcd data when the etcd step failed), so the node returns to the old version. The run exits with an error, and you investigate before retrying.
Why is kubeadm upgrade apply run on only one node per hop?
It upgrades the cluster-wide state: the stored kubeadm configuration, the `kubelet-config` ConfigMap, RBAC for bootstrap tokens and, eventually, the addons. The other nodes only need their local parts updated, which `kubeadm upgrade node` does by reading that already-updated cluster configuration.

saying these in an interview costs you the question

kubeadm upgrade apply also upgrades the kubelet binary
Run kubeadm upgrade apply on every control-plane node
Upgrade two of three stacked nodes at once to save time
kubeadm upgrade node needs the target version as an argument
CoreDNS is upgraded as soon as the first node finishes

On a three-node kubeadm cluster where every node runs the control plane, what is the sequence to upgrade Kubernetes by one minor version?

§ answer

basics

solid answer

code

§ go deeper

▾level notes4 levels

▾deep dive~3 min

The shape of the cluster

Step 1: the first control-plane node

Step 2: the kubelet on that node

Step 3: the other two nodes

Commands compared

Checks between nodes

▾common follow-ups2 questions

▾red flags5 flags

The shape of the cluster

Step 1: the first control-plane node

Step 2: the kubelet on that node

Step 3: the other two nodes

Commands compared

Checks between nodes

answer

go deeper