Distinguish two Kubernetes failures: a pod Pending with `pod has unbound immediate PersistentVolumeClaims`, versus a Deployment whose replica count never rises and for which no pod object is ever created. What causes each, and how do you confirm it?

## Two failures that look similar and are not Both present as "my workload is not running", but they sit on opposite sides of the scheduling boundary. Establishing which one you have is the first move: **does a pod object exist?** ``` kubectl get pods -n team-a ``` If a `Pending` pod is listed, the API accepted it and the scheduler is stuck. If nothing is listed, the pod was never created and the scheduler has never seen it. ## Case 1: the pod exists and its PVC is unbound A **PersistentVolumeClaim (PVC)** is a request for storage; a **PersistentVolume (PV)** is the actual storage. Binding is the matching of one to the other, either by finding a suitable pre-created PV or by **dynamic provisioning** through a StorageClass, which names a provisioner (a CSI driver) that creates the volume on demand. ### volumeBindingMode changes everything - **`Immediate`** — the volume is provisioned and bound as soon as the PVC is created, before any pod exists. The scheduler treats an unbound Immediate PVC as a hard blocker: `pod has unbound immediate PersistentVolumeClaims`. The problem is entirely in the storage layer. - **`WaitForFirstConsumer` (WFFC)** — binding is deliberately deferred until a pod that uses the PVC is being scheduled. The PVC sitting in `Pending` with `waiting for first consumer to be created` is **correct behaviour, not a fault**. WFFC exists because zonal storage cannot be moved: binding a volume in zone `a` before scheduling risks the scheduler wanting zone `b`, producing the classic `node(s) had volume node affinity conflict`. WFFC lets the scheduler choose a node first, then provisions the volume there. A very common misdiagnosis: someone sees a WFFC PVC `Pending` and chases the storage layer, when in fact the pod cannot be scheduled for an unrelated reason (insufficient CPU, a taint), so the volume is never provisioned. Read the **pod's** events, not the PVC's, in that case. ### Diagnosing an Immediate binding failure ``` kubectl get pvc -n team-a kubectl describe pvc data-api-0 -n team-a kubectl get storageclass ``` What the events tell you: - `no persistent volumes available for this claim and no storage class is set` — the PVC named no class and the cluster has no default class, or `storageClassName: ""` disabled dynamic provisioning and no matching static PV exists. - `storageclass.storage.k8s.io "fast-ssd" not found` — a typo or a class that exists only in another cluster. - `ProvisioningFailed` with a driver message — the CSI driver could not create the volume: cloud quota exhausted, wrong zone, IAM permissions missing, unsupported size or access mode. - Nothing at all — the CSI controller pods may be down; check them. Also check **access modes**: a `ReadWriteMany` claim against a block-storage provisioner that only supports `ReadWriteOnce` will never bind. And note that resizing and access-mode mismatches produce distinct events worth reading rather than guessing. ## Case 2: no pod object at all Here the failure is at **admission**, before the object is persisted. The controller that would create the pod records the rejection. ``` kubectl get deploy api -n team-a kubectl describe rs $(kubectl get rs -n team-a -l app=api -o name | tail -1) -n team-a kubectl get events -n team-a --sort-by=.lastTimestamp ``` The ReplicaSet event is `FailedCreate`, and the message names the cause: - **ResourceQuota** — `exceeded quota: team-a, requested: requests.cpu=2, used: requests.cpu=18, limited: requests.cpu=20`. A `ResourceQuota` caps a namespace's aggregate requests, limits, and object counts. Inspect it with `kubectl describe resourcequota -n team-a`, which prints Used versus Hard per resource. A critical wrinkle: **if a quota constrains a resource, every pod in that namespace must specify it** — a pod with no CPU request is rejected outright with `must specify requests.cpu`. This is why a namespace works fine until someone adds a quota and unrelated deployments start failing. - **LimitRange** — sets defaults and min/max per pod/container in the namespace. A container requesting more than the max, or less than the min, is rejected. LimitRange also *injects* defaults, which is the usual remedy for the quota wrinkle above. - **Pod Security admission** — `violates PodSecurity "restricted:latest"` when the pod requests privileges the namespace's policy forbids. - **Validating/mutating webhooks** — a policy engine (image-signature checks, required labels) denying the pod, or a webhook that is simply unreachable, which fails closed if `failurePolicy: Fail`. Note the asymmetry with Deployments: a Deployment's own creation succeeds because quota is enforced on **pods**, not on the Deployment object. So the Deployment exists and looks healthy while its ReplicaSet quietly fails to create pods — you must look one level down. ## The triage rule 1. Pod exists and is Pending → scheduler-side. Read the pod's events; if they point at volumes, move to the PVC and StorageClass. 2. PVC Pending with WFFC → check the *pod's* scheduling events first; the storage is waiting on placement. 3. No pod exists → admission-side. Describe the ReplicaSet (or Job/StatefulSet) for `FailedCreate` and read the quota/LimitRange/webhook message verbatim. Being able to state that boundary crisply — objects that exist versus objects that were never allowed to exist — is what the question is really testing.