A pod has been stuck in the `Pending` phase for ten minutes. Walk through how you find out why, and explain what a `FailedScheduling` event message such as "0/6 nodes are available: 4 Insufficient cpu, 2 node(s) had untolerated taint" is telling you.

## What Pending actually means A pod's `phase` is `Pending` from the moment the API server accepts it until all its containers are created. Two very different situations live in that window: 1. **Unscheduled** — `spec.nodeName` is empty. The scheduler has not found a node that satisfies the pod's constraints. 2. **Scheduled but not started** — a node is assigned and the kubelet is pulling images, attaching volumes, or waiting on an init container. The first check therefore separates the two: `kubectl get pod X -o wide` shows the NODE column, or `kubectl describe pod X` shows the `Node:` field. Empty means it is a scheduling problem and the events will come from the `default-scheduler`. Populated means it is a kubelet problem and you should look at image pulls and volume attachment instead. ## Reading the FailedScheduling event The scheduler works in two phases. **Filtering** runs a set of predicates against every node — does it have enough allocatable CPU/memory for the pod's requests, does the pod tolerate its taints, does it match nodeSelector/nodeAffinity, can the pod's volumes be attached there, do pod affinity/anti-affinity and topology spread constraints hold. Nodes that pass go to **scoring**, which ranks the survivors. If filtering leaves zero nodes, the pod stays Pending and the scheduler records a `FailedScheduling` event. That event's message is deliberately a **tally**: ``` 0/6 nodes are available: 4 Insufficient cpu, 2 node(s) had untolerated taint {workload=gpu: }. preemption: 0/6 nodes are available: 6 No preemption victims found for incoming pod. ``` - `0/6 nodes are available` — six nodes were considered, none survived filtering. - Each clause is `<count> <reason>`. The counts sum to the number of nodes considered (a node is reported under the first predicate that rejected it). - The trailing `preemption:` clause explains whether evicting lower-priority pods could have made room. This structure is the whole diagnostic value: it tells you not just *a* reason but the **distribution** of reasons, which is how you know whether one fix is enough. If five of six nodes say `Insufficient memory` and one says `untolerated taint`, lowering the memory request is likely sufficient. If the counts are split evenly across unrelated reasons, you need more than one change — or more nodes. ## The common reason strings | Message fragment | Meaning | |---|---| | `Insufficient cpu` / `Insufficient memory` | The pod's **requests** exceed the node's remaining allocatable | | `node(s) had untolerated taint {k=v}` | Node is tainted `NoSchedule`/`NoExecute`; the pod lacks a toleration | | `node(s) didn't match Pod's node affinity/selector` | `nodeSelector` or `nodeAffinity` excluded it | | `node(s) had volume node affinity conflict` | The pod's bound PersistentVolume lives in another zone | | `pod has unbound immediate PersistentVolumeClaims` | A PVC is not bound and cannot wait for scheduling | | `node(s) didn't match pod topology spread constraints` | Placing here would exceed `maxSkew` | | `node(s) didn't satisfy existing pods anti-affinity rules` | An anti-affinity rule forbids co-location | | `node(s) were unschedulable` | The node is cordoned (`SchedulingDisabled`) | ## When there is no event at all If `kubectl describe pod` shows an empty Events section, either the events have aged out (they default to a one-hour TTL) or the pod is not being processed by the scheduler you expect — check `spec.schedulerName` and whether the scheduler is running. If the *pod itself* never appears, the problem is upstream of scheduling: inspect the controller (`kubectl describe replicaset` / `deployment`) for `FailedCreate` from a ResourceQuota, LimitRange, or admission webhook. ## A repeatable triage order 1. `kubectl get pod -o wide` — is a node assigned? 2. `kubectl describe pod` — read the `FailedScheduling` tally. 3. Map each clause to a fix: requests too large, missing toleration, selector too narrow, PVC unbound, spread constraint too strict, nodes cordoned. 4. `kubectl describe node` on a representative candidate to confirm allocatable capacity and taints. 5. `kubectl get events --sort-by=.lastTimestamp` for the timeline, since the scheduler retries with exponential backoff and old messages can mislead. ## What to say in an interview Do not jump straight to "the cluster is out of resources". Say that `Pending` splits into unscheduled versus not-yet-started, that `describe pod` gives a per-reason node tally, and that the tally tells you whether one change fixes it. Naming three or four concrete reason strings and their fixes is what makes the answer credible.