What is a static pod in Kubernetes, how does it differ from a Pod created through the API server, and where are static pods actually used?
answer
- manifest dir /etc/kubernetes/manifests, kubelet watches it
- no scheduler, no controller, no RBAC on creation
- mirror pod is read-only; delete the file to remove
- kubeadm control plane = static pods (bootstrap paradox)
- for every-node agents use a DaemonSet instead
basics
~20 sA static pod is defined by a manifest file in a directory the kubelet watches (typically /etc/kubernetes/manifests). The kubelet runs it directly, with no scheduler or controller involved. It creates a read-only mirror Pod in the API so you can see it, but deleting that mirror does not stop it — you must remove the file.
solid answer
~50 sA **static pod** is a Pod the kubelet manages from a local manifest file rather than from the API server. The kubelet watches its static-pod path (usually `/etc/kubernetes/manifests`); any Pod YAML dropped there is started immediately, editing the file restarts it, deleting the file removes it. Differences from a normal Pod: - **No control plane involvement.** No scheduler binding, no ReplicaSet, no admission control or RBAC on its creation. It runs even if the API server is unreachable. - **Mirror Pod.** The kubelet creates a read-only representation in the API so `kubectl get pods` shows it, suffixed with the node name. `kubectl delete` on the mirror only makes the kubelet recreate it. - **Node-bound.** It never moves, and nothing reschedules it if the node dies. The canonical use is **bootstrapping the control plane**: kubeadm runs kube-apiserver, etcd, kube-scheduler and kube-controller-manager as static pods, solving the chicken-and-egg problem of needing an API server to schedule the API server. Node-level agents are better served by DaemonSets.
code
bash · 9 linesgrep staticPodPath /var/lib/kubelet/config.yaml
ls /etc/kubernetes/manifests/
# stop the API server temporarily (e.g. before an etcd restore)
mv /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/
# ...work...
mv /tmp/kube-apiserver.yaml /etc/kubernetes/manifests/
kubectl get pods -n kube-system -o wide | grep "$(hostname)"go deeper
Know that a static pod comes from a file on the node, is run by the kubelet alone, and is how the kubeadm control plane starts.
Explain mirror pods and why kubectl delete does not remove them, plus the concrete differences from a scheduled Pod.
Use them operationally — moving manifests to stop control-plane components during recovery — and argue DaemonSet versus static pod for node agents.
Treat write access to the static-pod path as a node-level privilege-escalation boundary, and reason about how much of the control plane should be self-hosted versus file-bootstrapped.
## Definition The kubelet has multiple pod sources. Besides its watch on the API server, it reads Pod manifests from a directory on the node's filesystem, configured by `staticPodPath` in the kubelet config (or the older `--pod-manifest-path` flag). Files there — one Pod per file, plain YAML or JSON — are **static pods**: the kubelet owns them entirely. The kubelet watches that directory. Add a file and the Pod starts within seconds; modify it and the kubelet restarts the Pod with the new spec; remove the file and the Pod is torn down. There is no `kubectl` in this loop at all. ## Mirror pods So operators can still see what is running, the kubelet creates a **mirror pod** in the API server for each static pod: a read-only object whose name is `<pod-name>-<node-name>`, owned by the Node and annotated as mirrored. Consequences: - `kubectl get pods -n kube-system` shows `kube-apiserver-node1`, `etcd-node1`, and so on. - `kubectl describe` and `kubectl logs` work, because logs come from the kubelet. - `kubectl delete pod kube-apiserver-node1` deletes only the mirror. The kubelet notices and recreates it — the underlying containers may restart, but the Pod does not go away. To stop it you move the manifest file. - `kubectl edit` on a mirror pod is pointless; the source of truth is the file. ## How it differs from an API-created Pod | Aspect | Static pod | Normal Pod | |---|---|---| | Source of truth | File on the node | etcd, via API server | | Scheduler | Not involved | Binds pod to node | | Controller (ReplicaSet etc.) | None | Usually owns the Pod | | Admission and RBAC on creation | Bypassed | Enforced | | Survives API-server outage | Yes | Existing pods run; new ones cannot be created | | Rescheduled on node failure | Never | Yes, by its controller | | Managed with | ssh / config management | kubectl / GitOps | That RBAC bypass is a security point worth making: write access to the static-pod directory (or to the kubelet config) is effectively root on the node and a way to run a privileged Pod without touching the API. It is a classic privilege-escalation path from a container that has a hostPath mount of the manifests directory. ## Why the control plane uses them A kubeadm-built control-plane node runs kube-apiserver, etcd, kube-controller-manager and kube-scheduler as static pods. This resolves a bootstrap paradox: to schedule the API server as a normal Pod you would need a working API server and scheduler. Static pods need neither — the kubelet reads a file and starts the containers. It also makes control-plane upgrades and repairs file-based: `kubeadm upgrade` rewrites the manifests, and a standard recovery move is to move a manifest out of the directory to stop a component (for example stopping the API server before an etcd restore) and move it back afterwards. ## When not to use them For anything you want on "every node" — log shippers, CNI agents, node exporters — use a **DaemonSet**. DaemonSets are declared once in the API, versioned in Git, rolled out with an update strategy, respect taints and tolerations, and are visible and manageable through normal tooling. Static pods require distributing files to every node with configuration management, offer no rollout control, and drift silently. Legitimate static-pod use cases are narrow: bootstrapping control-plane components, and node agents that must run *before* or *without* a functioning control plane. ## Operational notes - Static pods ignore nodeName, node selectors and affinity — they run where the file is, full stop. - They do not get ServiceAccount token projection the way normal Pods do; control-plane components use certificates and kubeconfig files mounted from hostPath. - Draining a node does not remove static pods; `kubectl drain` skips mirror pods since nothing could reschedule them. - If a static pod misbehaves, check the file first, then `journalctl -u kubelet` — the kubelet logs manifest parse errors that never reach the API.
- You run `kubectl delete pod etcd-node1` in kube-system. What happens?You delete only the mirror pod — the read-only API representation. The kubelet still holds the manifest file, notices the mirror is gone and recreates it; the underlying containers may restart but etcd is not removed from the node. The only way to stop it is to move or delete the etcd manifest in the static-pod directory on that node.
- When should you prefer a DaemonSet over static pods for a node-level agent?Almost always. A DaemonSet is declared once in the API, versioned in Git, rolled out with an update strategy, and respects taints, tolerations and node selectors, so it covers new nodes automatically. Static pods require distributing files to every node out of band and give no rollout or rollback control. Reserve static pods for components that must run before or without a working control plane.
A static pod is a note taped to the machine itself rather than an order filed with head office — the operator on site follows it even when head office is closed.
saying these in an interview costs you the question
- Believing kubectl delete removes a static pod
- Thinking the scheduler places static pods on nodes
- Using static pods where a DaemonSet is the right tool
- Not knowing the kubeadm control plane runs as static pods
- Assuming static pods are subject to admission control and RBAC