A pod sits in the status CreateContainerConfigError and never produces any application logs. What class of problem does that status indicate, and how is it different from a container that starts and then crashes?

## Where it sits in the container startup sequence Starting a container is a pipeline, and each stage has its own failure status: 1. **Schedule the pod** onto a node — failure: pod stays `Pending` with `FailedScheduling`. 2. **Mount volumes** — failure: `FailedMount`, pod stuck in `ContainerCreating`. 3. **Pull the image** — failure: `ErrImagePull` → `ImagePullBackOff`. 4. **Build the container configuration** (env, mounts, security context) — failure: **`CreateContainerConfigError`**. 5. **Create the container** in the runtime — failure: `CreateContainerError`. 6. **Start the process** — failure: `RunContainerError`. 7. **Process runs, then exits** — failure: `CrashLoopBackOff` with an exit code. Knowing the position tells you immediately what evidence exists. Anything at stage 4 or earlier means no process ever ran, so `kubectl logs` cannot help and there is no exit code to interpret. All the information is in `kubectl describe pod`'s Events and container status message. ## What actually triggers it By far the most common causes are unresolved configuration references: - **Missing ConfigMap or Secret.** `envFrom: [{configMapRef: {name: app-config}}]` where `app-config` does not exist in the pod's namespace. Message: `configmap "app-config" not found`. - **Missing key.** The object exists but the specific key does not: `env: [{valueFrom: {secretKeyRef: {name: app-secrets, key: DATABASE_URL}}}]`. Message: `couldn't find key DATABASE_URL in Secret default/app-secrets`. - **Wrong namespace.** ConfigMaps and Secrets are namespace-scoped and cannot be referenced across namespaces. A manifest copied between namespaces fails here. - **Invalid content for the field.** For example a `securityContext.runAsUser` that conflicts with the image's `USER` under a `runAsNonRoot: true` policy surfaces at container creation rather than as a crash. A related but distinct failure is a ConfigMap or Secret used as a *volume* that does not exist: that typically hangs at `ContainerCreating` with a `FailedMount` event instead, because the failure happens during volume setup rather than config assembly. ## Why it is so often a race, not a bug The classic scenario is applying a Deployment and its ConfigMap in the same change, with the Deployment landing first — or a Secret that a CI pipeline or an external-secrets operator has not synced yet. The pods come up, cannot resolve the reference, and sit in `CreateContainerConfigError`. Nothing is permanently broken: the kubelet retries with backoff, and the moment the missing object appears the container is configured and started. No pod deletion is required. This is also why the status is a useful signal in GitOps setups: it tells you an ordering or sync dependency exists that your delivery tooling is not honouring. Server-side apply of a whole manifest set, or a sync-wave/dependency ordering in the delivery tool, removes the window. ## Optional references Both `configMapKeyRef`/`secretKeyRef` and `envFrom` support `optional: true`. With it, a missing source or key is silently skipped and the container starts with the variable unset. That is right for genuinely optional feature flags and wrong for anything the application needs to function — an app that starts with a missing `DATABASE_URL` and then crashes at first query is harder to diagnose than one that never starts with a message naming the exact key. Use `optional` deliberately, not to make an error message go away. ## Diagnosing it ``` kubectl describe pod <pod> | sed -n '/Events/,$p' kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[0].state.waiting.message}' ``` The message names the object and, when relevant, the key. Then verify existence and content in the *pod's* namespace: ``` kubectl get configmap app-config -n team-a kubectl get secret app-secrets -n team-a -o jsonpath='{.data}' | tr ',' '\n' ``` Note that key names are case-sensitive and that Secret keys must be valid environment variable names when consumed via `envFrom` — invalid names are skipped (with an event) rather than injected. ## The wider habit When triaging any stuck pod, first classify it by which stage of the pipeline it died at, because that determines which tool gives you evidence. `Pending` → scheduling; `ContainerCreating` → volumes/network; `ImagePullBackOff` → registry; `CreateContainerConfigError` → missing config objects; `CrashLoopBackOff` → the application itself, and only then are logs and exit codes meaningful.