You need a root shell on a Kubernetes worker node itself — to check the kubelet, disk usage, or the container runtime — but SSH to nodes is not available. How do you get one with kubectl, and what does it give you?

seniorshould knowfreq 33%

answer

kubectl debug node/<name>
hostNetwork + hostPID + hostIPC
node root at /host, then chroot /host
journalctl kubelet, crictl, df -h, dmesg
privileged namespace + manual delete

basics

~20 s

kubectl debug node/NODE -it --image=busybox creates a privileged pod pinned to that node in the host network, PID and IPC namespaces, with the node's root filesystem mounted at /host. Run chroot /host to work as if on the node. Delete the pod afterwards.

solid answer

~50 s

kubectl debug node/ip-10-0-1-7 -it --image=busybox schedules a pod named node-debugger-<node>-<suffix> onto that specific node with hostNetwork, hostPID and hostIPC enabled and the node's root filesystem bind-mounted at /host. chroot /host then gives you effectively a node shell: journalctl -u kubelet, crictl ps, df -h /var/lib/kubelet, dmesg, ip a, iptables-save.

Because hostPID is set you can see every process on the node, and hostNetwork means you are on the node's network stack, not a pod's.

The caveats are serious. This is a privileged pod, so the namespace must permit it under Pod Security admission — a restricted namespace will reject it. It is fully audited, it consumes node resources, and kubectl does not clean it up: kubectl delete pod node-debugger-... when done. Access to it is equivalent to root on the node, so treat the permission as cluster-admin-grade.

`kubectl debug node/ip-10-0-1-7 -it --image=busybox` schedules a pod named `node-debugger-<node>-<suffix>` onto that specific node with `hostNetwork`, `hostPID` and `hostIPC` enabled and the node's root filesystem bind-mounted at `/host`. `chroot /host` then gives you effectively a node shell: `journalctl -u kubelet`, `crictl ps`, `df -h /var/lib/kubelet`, `dmesg`, `ip a`, `iptables-save`. Because `hostPID` is set you can see every process on the node, and `hostNetwork` means you are on the node's network stack, not a pod's. The caveats are serious. This is a privileged pod, so the namespace must permit it under Pod Security admission — a `restricted` namespace will reject it. It is fully audited, it consumes node resources, and kubectl does **not** clean it up: `kubectl delete pod node-debugger-...` when done. Access to it is equivalent to root on the node, so treat the permission as cluster-admin-grade.

code

bash · 9 lines

bash

kubectl debug node/ip-10-0-1-7 -it --image=busybox:1.36
# inside the debugger pod:
chroot /host
journalctl -u kubelet -n 200 --no-pager
crictl ps -a | head
df -h /var/lib/containerd /var/log
dmesg -T | grep -i -E 'oom|blocked'
exit; exit
kubectl delete pod node-debugger-ip-10-0-1-7-xxxxx

go deeper

for a junior

Awareness is enough: there is a kubectl way onto a node when SSH is unavailable, and it is privileged.

for a middle

Know the command, that the node root is at /host, and the chroot step, plus that cleanup is manual.

for a senior

Explain which node-level failures justify it (kubelet, disk, runtime, kernel OOM, conntrack) and how it interacts with Pod Security admission and audit.

for a principal

Treat the capability as cluster-admin-equivalent: decide who holds it, how it is time-boxed and reviewed, and what node-level telemetry removes the need for it.

Why this exists

Some failures are not inside a container: the kubelet is unhealthy, the disk holding images is full, the container runtime is wedged, the node's clock is wrong, conntrack is exhausted, or kernel messages show OOM kills. Managed clusters often have no SSH path to nodes at all. kubectl debug node/<name> gives an in-cluster route to node-level state.

What the command builds

kubectl debug node/<node> -it --image=<image> creates an ordinary pod, not an ephemeral container, with:

spec.nodeName pinned to the target node, bypassing normal scheduling (so it lands even on a cordoned node),
hostNetwork: true, hostPID: true, hostIPC: true,
a hostPath volume for / mounted at /host,
an elevated security context,
and tolerations broad enough to land on tainted nodes.

It is created in your current namespace and named node-debugger-<node>-<random>. chroot /host switches your root to the node's filesystem so the node's own binaries and configuration are in scope — without it you are running the debug image's tools against /host/... paths.

What you can actually inspect

Kubelet and runtime health: journalctl -u kubelet -n 200, systemctl status containerd, crictl ps -a, crictl logs <id>.
Disk and inodes: df -h, df -i on /var/lib/containerd, /var/lib/kubelet and /var/log — the usual cause of DiskPressure and image-pull failures.
Kernel evidence: dmesg -T | grep -i oom for kernel OOM kills that never reach pod events.
Networking: ip a, ip route, conntrack -S, iptables-save for CNI and service-proxy rules, and ss -lntp on the host stack.
On-disk pod state: /host/var/log/pods/<ns>_<pod>_<uid>/<container>/ holds the raw container logs the kubelet serves.

Security and hygiene

This is root on the node, full stop. From that shell you can read every container's filesystem, every mounted secret and every service-account token on the node, and modify the host. Practical consequences:

Pod Security admission matters. A namespace enforcing baseline or restricted will reject the debugger pod; teams typically keep one namespace labelled privileged for this and grant pod-create there narrowly.
RBAC. Granting create pods in that namespace is effectively granting node root, and by extension a path to cluster-admin. It belongs in a break-glass role, not a default on-call role.
Audit. The pod creation appears in the API server audit log, and the pod itself remains visible — good for review, bad for tidiness.
Cleanup is manual. kubectl leaves the pod behind after you exit; forgotten debugger pods are a recurring finding in cluster reviews. Delete it, and prefer a small image that is already present or quick to pull — on a node with a broken runtime or a full disk the image pull itself may fail, at which point node access must come from the cloud provider's console or SSH.

Choosing between the three debug modes

kubectl debug covers three different situations and interviewers like to hear them separated: ephemeral container attached to a live pod (kubectl debug -it pod/x --image=... --target=c) for inspecting a running workload; pod copy (--copy-to) when the container will not stay up or needs a changed command; node debugger (node/<name>) when the problem is the machine rather than the workload. Reaching for the node debugger first is a red flag — it is the most privileged option and usually unnecessary.

## Why this exists Some failures are not inside a container: the kubelet is unhealthy, the disk holding images is full, the container runtime is wedged, the node's clock is wrong, conntrack is exhausted, or kernel messages show OOM kills. Managed clusters often have no SSH path to nodes at all. `kubectl debug node/<name>` gives an in-cluster route to node-level state. ## What the command builds `kubectl debug node/<node> -it --image=<image>` creates an ordinary pod, not an ephemeral container, with: - `spec.nodeName` pinned to the target node, bypassing normal scheduling (so it lands even on a cordoned node), - `hostNetwork: true`, `hostPID: true`, `hostIPC: true`, - a `hostPath` volume for `/` mounted at `/host`, - an elevated security context, - and tolerations broad enough to land on tainted nodes. It is created in your current namespace and named `node-debugger-<node>-<random>`. `chroot /host` switches your root to the node's filesystem so the node's own binaries and configuration are in scope — without it you are running the debug image's tools against `/host/...` paths. ## What you can actually inspect - **Kubelet and runtime health**: `journalctl -u kubelet -n 200`, `systemctl status containerd`, `crictl ps -a`, `crictl logs <id>`. - **Disk and inodes**: `df -h`, `df -i` on `/var/lib/containerd`, `/var/lib/kubelet` and `/var/log` — the usual cause of DiskPressure and image-pull failures. - **Kernel evidence**: `dmesg -T | grep -i oom` for kernel OOM kills that never reach pod events. - **Networking**: `ip a`, `ip route`, `conntrack -S`, `iptables-save` for CNI and service-proxy rules, and `ss -lntp` on the host stack. - **On-disk pod state**: `/host/var/log/pods/<ns>_<pod>_<uid>/<container>/` holds the raw container logs the kubelet serves. ## Security and hygiene This is root on the node, full stop. From that shell you can read every container's filesystem, every mounted secret and every service-account token on the node, and modify the host. Practical consequences: - **Pod Security admission matters.** A namespace enforcing `baseline` or `restricted` will reject the debugger pod; teams typically keep one namespace labelled `privileged` for this and grant pod-create there narrowly. - **RBAC.** Granting `create pods` in that namespace is effectively granting node root, and by extension a path to cluster-admin. It belongs in a break-glass role, not a default on-call role. - **Audit.** The pod creation appears in the API server audit log, and the pod itself remains visible — good for review, bad for tidiness. - **Cleanup is manual.** kubectl leaves the pod behind after you exit; forgotten debugger pods are a recurring finding in cluster reviews. Delete it, and prefer a small image that is already present or quick to pull — on a node with a broken runtime or a full disk the image pull itself may fail, at which point node access must come from the cloud provider's console or SSH. ## Choosing between the three debug modes `kubectl debug` covers three different situations and interviewers like to hear them separated: **ephemeral container** attached to a live pod (`kubectl debug -it pod/x --image=... --target=c`) for inspecting a running workload; **pod copy** (`--copy-to`) when the container will not stay up or needs a changed command; **node debugger** (`node/<name>`) when the problem is the machine rather than the workload. Reaching for the node debugger first is a red flag — it is the most privileged option and usually unnecessary.

Why is granting a team the ability to run kubectl debug node/ effectively the same as giving them cluster-admin?
The debugger pod runs privileged in the host namespaces with the node filesystem mounted, so its user can read every service-account token and secret mounted on that node, including tokens of highly privileged workloads, and can modify the kubelet's configuration or static pod manifests. Anyone with that access can escalate to control-plane-equivalent privileges, so it must be a time-boxed break-glass grant, not a standing permission.

saying these in an interview costs you the question

Calling it an ephemeral container — node debugging creates a normal privileged pod
Forgetting chroot /host and then concluding the node has no kubelet logs
Leaving debugger pods running on nodes after the incident
Assuming it works in any namespace regardless of Pod Security admission
Reaching for node debug for an application problem that pod-level tools would answer

You need a root shell on a Kubernetes worker node itself — to check the kubelet, disk usage, or the container runtime — but SSH to nodes is not available. How do you get one with kubectl, and what does it give you?

§ answer

basics

solid answer

code

§ go deeper

▾level notes4 levels

▾deep dive~3 min

Why this exists

What the command builds

What you can actually inspect

Security and hygiene

Choosing between the three debug modes

▾common follow-ups1 question

▾red flags5 flags

§ context

▾related questions4 questions

▾related topics3 topics

Why this exists

What the command builds

What you can actually inspect

Security and hygiene

Choosing between the three debug modes

answer

go deeper

context