Troubleshooting

Map the failing pods to their nodes, then read each pod's Failed event for the actual pull error. The error class (not found, 401/403, 429, network or TLS) combined with what differs about those nodes (cache, egress IP, pool, credentials) explains the split.

They come from metrics-server, an optional add-on that scrapes each kubelet every ~15 seconds and keeps only the latest sample in memory. CPU is an average rate over that window and memory is working-set bytes — so no history, no spikes, and not the same as your app's heap.

Events are ordinary API objects stored in etcd with a lease set by kube-apiserver's --event-ttl, default one hour, renewed whenever the Event is updated. After that they vanish, so keeping them means exporting them continuously to a durable store.

Ephemeral storage is the container writable layer, emptyDir volumes, and container logs on the node's disk. The kubelet evicts pods when nodefs/imagefs run low, or when a pod exceeds its ephemeral-storage limit. Fix with log rotation, sized emptyDir, ephemeral-storage requests/limits, and image garbage collection.

The node controller adds a NoExecute taint (node.kubernetes.io/not-ready or unreachable). Pods carry a default toleration with tolerationSeconds 300, so after 5 minutes the taint manager deletes them and controllers recreate them elsewhere. Kubelet node-pressure eviction is different: local, resource-driven, ranked by QoS.

An unbound PVC means storage was never provisioned — check the PVC events and StorageClass. A missing pod entirely means admission rejected creation: describe the ReplicaSet for FailedCreate naming a ResourceQuota, LimitRange, or webhook. One is a scheduling problem; the other happens before scheduling.

A topologySpreadConstraint limits how uneven pod placement may be across a topology key (zone, node): the count difference between domains must stay within maxSkew. With whenUnsatisfiable: DoNotSchedule, a pod that would break the skew stays Pending. Relax to ScheduleAnyway, raise maxSkew, or add capacity in the short domains.

A silent kill after a healthy period usually means the kubelet is killing the container because its liveness probe failed. Check kubectl describe pod for Unhealthy events and the probe's path, port, timeout and thresholds, then call the same endpoint from inside the pod.

Endpoint updates are eventually consistent: a terminating pod stops accepting connections before every node's kube-proxy has removed it, so a fraction of requests are sent to a dead backend. Fix with a preStop sleep, graceful shutdown, and readiness flipping before termination.

Silent drops point at NetworkPolicy. List policies in both namespaces and see which select the client and server pods: once any policy selects a pod for a direction, everything not explicitly allowed in that direction is denied. Check ingress and egress separately, and confirm the CNI enforces policy at all.

First match the service's parallelism to its CPU limit, since most throttling below the limit is over-parallelism. Raise the limit when real demand exceeds it. Remove it only with an accurate request and an accepted loss of Guaranteed QoS.

Keep images minimal and move debugging out of them: an approved toolbox image attached as an ephemeral container with kubectl debug --target, granted through a time-boxed break-glass role, audited, and paired with in-app diagnostics so most incidents need no shell at all.

kubectl cp is exec plus tar: it runs tar inside the container and streams the archive over the exec channel. If the image has no tar binary — distroless or scratch — the copy fails. Fall back to exec with cat, or a debug container.

Working set is the cgroup's memory usage minus inactive file cache, so it still counts active page cache the kernel can reclaim. RSS counts anonymous memory that cannot be reclaimed without swap. RSS near the limit predicts an OOM kill.

A webhook with failurePolicy Fail intercepts pod creation, and its backend has no running pods, so every create is rejected, including creates of its replacement pods. Break the loop by patching the webhook configuration to Ignore or narrowing its scope.

Make failures non-destructive and detectable: readiness-gated rolling updates with a progress deadline and automatic rollback, digest-pinned images served from a registry mirror, admission rules requiring probes and resource limits, and alerts on restart and pull-failure rates rather than on individual pods.