A newly scheduled pod stays in ContainerCreating and its events show a failure to set up the sandbox network. How would you diagnose it, and what are the likely causes?

seniorshould knowfreq 42%

answer

no app container started → look at node, not logs
one node vs all nodes = first split
/etc/cni/net.d config + /opt/cni/bin binaries
node podCIDR assigned? pool exhausted?
stale IPAM entries from failed DEL

basics

~20 s

Read the pod events and note the node, then check that node: is the network plugin's DaemonSet pod healthy there, is a valid config present in /etc/cni/net.d, are the binaries in /opt/cni/bin, does the node have a pod CIDR assigned, and is its address pool exhausted or full of stale allocations?

solid answer

~50 s

The pod has no application containers yet — the runtime failed at sandbox setup — so container logs are useless. Work node-first.

kubectl describe pod for the exact plugin error text and the node name.
Is it one node or all nodes? One node points at that node's plugin state; all nodes point at the plugin itself or at the control plane.
On the node: is the plugin's DaemonSet pod Running there? Is there a valid *.conflist in /etc/cni/net.d and are the named binaries in /opt/cni/bin? A node that is NotReady with a "network plugin not ready" message usually just lacks that file.
Check IPAM: the node's pod CIDR may be unassigned, its pool exhausted, or the allocation store may be full of stale entries leaked by failed teardowns.
Check the runtime and kubelet logs for the plugin's stderr, and confirm plugin/binary versions match after an upgrade.

The pod has no application containers yet — the runtime failed at sandbox setup — so container logs are useless. Work node-first. 1. `kubectl describe pod` for the exact plugin error text and the node name. 2. Is it **one node or all nodes**? One node points at that node's plugin state; all nodes point at the plugin itself or at the control plane. 3. On the node: is the plugin's DaemonSet pod Running there? Is there a valid `*.conflist` in `/etc/cni/net.d` and are the named binaries in `/opt/cni/bin`? A node that is `NotReady` with a "network plugin not ready" message usually just lacks that file. 4. Check IPAM: the node's pod CIDR may be unassigned, its pool exhausted, or the allocation store may be full of stale entries leaked by failed teardowns. 5. Check the runtime and kubelet logs for the plugin's stderr, and confirm plugin/binary versions match after an upgrade.

code

bash · 11 lines

bash

kubectl describe pod api-0 | sed -n '/Events/,$p'
kubectl get pods -A -o wide | grep ContainerCreating   # one node or many?

kubectl get node node-b -o jsonpath='{.spec.podCIDR}{"\n"}'
kubectl -n kube-system get pods -o wide --field-selector spec.nodeName=node-b
kubectl -n kube-system logs <cni-agent-pod-on-node-b> --tail=100

# on the node
ls -l /etc/cni/net.d/ /opt/cni/bin/
journalctl -u containerd -n 200 --no-pager | grep -i cni
ls /var/lib/cni/networks/*/ | wc -l    # allocations vs actual pods on this node

go deeper

for a junior

Know that the pod has no containers yet, that the events hold the real error, and that the network plugin on that node is the thing to check.

for a middle

Give an ordered checklist — events, node, plugin pod, config and binaries, pod CIDR — and explain why container logs are empty.

for a senior

Lead with blast-radius triage, distinguish exhaustion from leaked allocations with evidence, and name the durable fixes: address-availability alerts, canary node for plugin upgrades.

for a principal

Discuss the systemic angle: the plugin is a startup-path dependency for every pod, so its upgrade process, its dependency on the API server, and address-capacity headroom belong in the platform's risk model.

Why the symptom is shaped this way

During pod startup, the CRI runtime creates the pod sandbox and its empty network namespace, then invokes the network plugin against it. Only if the plugin returns success does the runtime start the application containers. So a plugin failure produces a pod that is scheduled, has a node, and never gets an IP or a running container. kubectl logs returns nothing useful because nothing of yours has run. All the evidence is in events and on the node.

Step 1 — read the error verbatim

kubectl describe pod <name> surfaces the message the runtime got back, and plugins put real detail there: no IP addresses available in range, plugin not found, unable to connect to the plugin's local agent, invalid config. Do not paraphrase it away; the text usually names the failing stage.

Step 2 — establish the blast radius

Ask immediately whether other pods are failing and where.

One node only → node-local state: a crashed plugin agent, a missing or corrupt config file, exhausted address pool on that node, a missing pod CIDR assignment, disk-full preventing the plugin from writing its store.
Every node, all new pods → the plugin's control plane, a bad rollout of its DaemonSet, an incompatible upgrade, or a control-plane issue such as pod CIDRs not being allocated at all.
Only pods in one namespace or with one annotation → something workload-specific: a policy or admission mutation, a requested extra network, a pool selector.

A quick kubectl get pods -A -o wide | grep ContainerCreating answers this in one command.

Step 3 — the node checklist

On the affected node (via a debug pod or SSH):

Plugin agent: is the plugin's DaemonSet pod Running and recently restarted? Its logs are the single best source. Many plugins write their config file on agent startup, so a crash-looping agent and a missing config are the same incident.
Config: /etc/cni/net.d/ should contain a valid *.conflist. Watch for two configs where the lexically first is a leftover from a previous plugin — the runtime picks that one and you get an unexpected datapath or a hard failure. An empty directory is why nodes report themselves not ready for networking.
Binaries: every type named in the config must exist in /opt/cni/bin, executable, and built for a spec version the runtime supports. "plugin not found" after an upgrade is common when a new config references a binary the old DaemonSet never installed.
Runtime/kubelet logs: journalctl -u containerd and -u kubelet carry the plugin's stderr and exit codes.

Step 4 — IPAM problems, the most common real cause

Three distinct failures look alike:

No pod CIDR assigned to the node. If the controller manager is not allocating node CIDRs, or the cluster CIDR is fully carved up among existing nodes, a new node gets nothing and every pod on it fails. Check kubectl get node <n> -o jsonpath='{.spec.podCIDR}'.
Pool exhausted. The node's range is genuinely full — for example a /24 slice against a high pod-density node, or a provider plugin that has hit the instance's address limit.
Leaked allocations. Teardown (DEL) failed in the past and the IPAM store still records addresses for containers that no longer exist. The node looks nearly idle but reports no addresses available. With the reference host-local IPAM the state lives under /var/lib/cni/networks/<network>/, one file per address naming its container; entries whose container is gone are the leaks. Newer plugin versions garbage-collect this; older ones need the agent restarted or stale entries cleared.

Distinguish them by comparing running pods on the node against allocated addresses. If allocations far exceed pods, it is leakage; if they match and the range is small, it is genuine exhaustion and the fix is a bigger per-node block or lower pod density.

Step 5 — the less common causes

Version skew after upgrading the plugin: config cniVersion newer than the binaries or the runtime supports.
Sysctl/module prerequisites missing after a node image change — a required kernel module or forwarding sysctl that the plugin expects.
The plugin's own dependencies: a plugin whose agent needs the API server or its datastore will fail sandbox setup while that is unreachable, which can turn a control-plane blip into an inability to start pods.
A second network plugin left installed, fighting over the config directory.

Closing the loop

After the fix, most stuck pods recover on their own because the runtime retries sandbox creation; delete the ones that do not. The durable follow-ups are the ones that prevent recurrence: alert on nodes reporting no available addresses, monitor plugin agent restarts, and treat plugin upgrades as datapath changes — rolled node by node, with a canary node validated before the fleet.

## Why the symptom is shaped this way During pod startup, the CRI runtime creates the pod sandbox and its empty network namespace, then invokes the network plugin against it. Only if the plugin returns success does the runtime start the application containers. So a plugin failure produces a pod that is scheduled, has a node, and never gets an IP or a running container. `kubectl logs` returns nothing useful because nothing of yours has run. All the evidence is in events and on the node. ## Step 1 — read the error verbatim `kubectl describe pod <name>` surfaces the message the runtime got back, and plugins put real detail there: no IP addresses available in range, plugin not found, unable to connect to the plugin's local agent, invalid config. Do not paraphrase it away; the text usually names the failing stage. ## Step 2 — establish the blast radius Ask immediately whether other pods are failing and where. - **One node only** → node-local state: a crashed plugin agent, a missing or corrupt config file, exhausted address pool on that node, a missing pod CIDR assignment, disk-full preventing the plugin from writing its store. - **Every node, all new pods** → the plugin's control plane, a bad rollout of its DaemonSet, an incompatible upgrade, or a control-plane issue such as pod CIDRs not being allocated at all. - **Only pods in one namespace or with one annotation** → something workload-specific: a policy or admission mutation, a requested extra network, a pool selector. A quick `kubectl get pods -A -o wide | grep ContainerCreating` answers this in one command. ## Step 3 — the node checklist On the affected node (via a debug pod or SSH): - **Plugin agent**: is the plugin's DaemonSet pod `Running` and recently restarted? Its logs are the single best source. Many plugins write their config file *on agent startup*, so a crash-looping agent and a missing config are the same incident. - **Config**: `/etc/cni/net.d/` should contain a valid `*.conflist`. Watch for two configs where the lexically first is a leftover from a previous plugin — the runtime picks that one and you get an unexpected datapath or a hard failure. An empty directory is why nodes report themselves not ready for networking. - **Binaries**: every `type` named in the config must exist in `/opt/cni/bin`, executable, and built for a spec version the runtime supports. "plugin not found" after an upgrade is common when a new config references a binary the old DaemonSet never installed. - **Runtime/kubelet logs**: `journalctl -u containerd` and `-u kubelet` carry the plugin's stderr and exit codes. ## Step 4 — IPAM problems, the most common real cause Three distinct failures look alike: 1. **No pod CIDR assigned to the node.** If the controller manager is not allocating node CIDRs, or the cluster CIDR is fully carved up among existing nodes, a new node gets nothing and every pod on it fails. Check `kubectl get node <n> -o jsonpath='{.spec.podCIDR}'`. 2. **Pool exhausted.** The node's range is genuinely full — for example a `/24` slice against a high pod-density node, or a provider plugin that has hit the instance's address limit. 3. **Leaked allocations.** Teardown (`DEL`) failed in the past and the IPAM store still records addresses for containers that no longer exist. The node looks nearly idle but reports no addresses available. With the reference `host-local` IPAM the state lives under `/var/lib/cni/networks/<network>/`, one file per address naming its container; entries whose container is gone are the leaks. Newer plugin versions garbage-collect this; older ones need the agent restarted or stale entries cleared. Distinguish them by comparing running pods on the node against allocated addresses. If allocations far exceed pods, it is leakage; if they match and the range is small, it is genuine exhaustion and the fix is a bigger per-node block or lower pod density. ## Step 5 — the less common causes - **Version skew** after upgrading the plugin: config `cniVersion` newer than the binaries or the runtime supports. - **Sysctl/module prerequisites** missing after a node image change — a required kernel module or forwarding sysctl that the plugin expects. - **The plugin's own dependencies**: a plugin whose agent needs the API server or its datastore will fail sandbox setup while that is unreachable, which can turn a control-plane blip into an inability to start pods. - **A second network plugin** left installed, fighting over the config directory. ## Closing the loop After the fix, most stuck pods recover on their own because the runtime retries sandbox creation; delete the ones that do not. The durable follow-ups are the ones that prevent recurrence: alert on nodes reporting no available addresses, monitor plugin agent restarts, and treat plugin upgrades as datapath changes — rolled node by node, with a canary node validated before the fleet.

The node hosts only a handful of pods but every new pod fails with no addresses available in its range. What happened?
The IPAM store has leaked allocations. Each failed or interrupted teardown left the address marked in use, so the recorded allocations far exceed the pods actually running. Compare the entries in the IPAM state directory against the node's running pods; entries naming containers that no longer exist are the leaks. Reclaim them via the plugin's garbage collection or by clearing stale entries, then fix the teardown failure so it does not recur.
Every new pod cluster-wide fails at sandbox setup right after you upgraded the network plugin. Where do you look first?
At version skew between the newly written config and what is installed. Check whether the DaemonSet rolled out everywhere, whether the config in /etc/cni/net.d now names a plugin binary that is not present in /opt/cni/bin, and whether its cniVersion exceeds what the container runtime supports. Roll back the DaemonSet to restore pod creation, then re-attempt on a single canary node.

saying these in an interview costs you the question

Hunting in kubectl logs when no application container has started
Deleting and recreating the pod repeatedly instead of finding the node-local cause
Ignoring whether the failure is one node or the whole cluster
Assuming exhaustion whenever addresses run out, without checking for leaked allocations
Forgetting that many plugins write the CNI config only when their agent starts, so a crash-looping agent explains a missing config

A newly scheduled pod stays in ContainerCreating and its events show a failure to set up the sandbox network. How would you diagnose it, and what are the likely causes?

§ answer

basics

solid answer

code

§ go deeper

▾level notes4 levels

▾deep dive~4 min

Why the symptom is shaped this way

Step 1 — read the error verbatim

Step 2 — establish the blast radius

Step 3 — the node checklist

Step 4 — IPAM problems, the most common real cause

Step 5 — the less common causes

Closing the loop

▾common follow-ups2 questions

▾red flags5 flags

§ context

▾related questions5 questions

▾related topics3 topics

Why the symptom is shaped this way

Step 1 — read the error verbatim

Step 2 — establish the blast radius

Step 3 — the node checklist

Step 4 — IPAM problems, the most common real cause

Step 5 — the less common causes

Closing the loop

answer

go deeper

context