Troubleshooting

Use kubectl exec -it POD -c CONTAINER -- sh. It starts an extra process inside an already-running container, streamed through the API server and the kubelet. The binary must exist in the image and the container must be running.

imagePullPolicy tells the kubelet when to check the registry: Always on every start, IfNotPresent only when the image is not cached, Never not at all. Unset, it defaults to Always for :latest or images with no tag and no digest, otherwise IfNotPresent.

kubectl logs POD -c CONTAINER shows the current instance; --previous shows the last terminated one. Add -f to follow, --since and --tail to bound output. Nothing appears if the app logs to a file instead of stdout/stderr, the pod never started, or the logs rotated.

No. Kubernetes keeps container logs only as files on the pod's node, tied to that pod; once the pod is deleted the kubelet removes its containers and log directory, and the replacement pod starts with empty logs.

Evicted means the kubelet killed the pod to reclaim a scarce node resource (memory or disk). The pod object stays as a dead record. Check kubectl describe pod for the reason message and kubectl describe node for MemoryPressure or DiskPressure conditions.

Pending means the pod is accepted but not bound to a node (or images are still pulling). Run kubectl describe pod and read the Events section: the scheduler's FailedScheduling message lists, per rejection reason, how many nodes it disqualified — that count tells you which constraint to fix.

The kubelet cannot fetch the image. Usual causes: wrong repository name, tag or digest; the tag does not exist; a private registry with missing or wrong imagePullSecrets; registry unreachable or rate-limiting. Run kubectl describe pod and read the exact pull error in Events.

port is what the Service listens on at its cluster IP; targetPort is the pod port traffic is forwarded to; nodePort is an extra port opened on every node for NodePort/LoadBalancer Services; containerPort is documentation of what the container serves. A wrong targetPort gives endpoints but connection refused.

Connection refused means nothing is listening, so the API server is down or the address is wrong. An x509 expiry means the TLS handshake failed, usually on an expired serving certificate. TooManyRequests (HTTP 429) means a live API server is rejecting load.

kubectl debug -it POD --image=busybox --target=app adds an ephemeral container to the live pod, giving you tools without a restart. It shares the pod network; --target additionally shares the target container's process namespace so you see its processes and files via /proc.

A pod's imagePullSecrets names Secrets of type kubernetes.io/dockerconfigjson in its own namespace. The kubelet reads them and passes matching registry credentials to the runtime. Because the reference is namespace-local, a Secret in another namespace is never found.

describe is a human summary that also joins in Events referencing the object; -o yaml is the exact API object, spec plus status, suitable for scripting. Events are separate namespaced objects, expire after about an hour, are aggregated with counts, and are not sorted by default.

Confirm context and namespace, then kubectl get pods -o wide for phase, readiness and restarts; kubectl describe pod for events and container state; kubectl logs (with --previous if it restarted); kubectl get -o yaml for exact status; then widen to events, the controller, endpoints, nodes and kubectl top.

The kubelet watches signals like memory.available and nodefs.available. Crossing a hard threshold evicts pods immediately with no grace period; a soft threshold must stay crossed for a configured grace period first, and pods get a bounded termination grace period. Crossing sets node conditions MemoryPressure or DiskPressure.

The kubelet ranks pods by whether usage exceeds requests, then by pod priority, then by how much usage exceeds requests. In practice BestEffort pods (no requests) go first, then Burstable pods over their request, and Guaranteed pods (limits equal requests) last.

The scheduler sums pods' CPU/memory requests, not actual usage, against the node's allocatable (capacity minus kube-reserved, system-reserved and eviction thresholds). A cluster idle at 30% can be fully reserved if requests are inflated. Fix by right-sizing requests or adding capacity.

Taints are set on nodes to repel pods; a pod needs a matching toleration to be allowed there. nodeSelector/nodeAffinity are set on the pod to require node labels. Taints repel, selectors attract — you usually need both: add the toleration and make sure node labels match.

It means the container keeps exiting and the kubelet is now waiting, with a growing delay, before restarting it again. It is a symptom, not a cause. Use kubectl describe pod for the last exit code and reason, then kubectl logs --previous for the dead container's output.

The container's processes exceeded the memory limit in its cgroup, so the kernel's OOM killer sent SIGKILL — exit 137 is 128+9. Fix by measuring real memory use, then either raising the limit or making the runtime respect it (for example JVM heap sizing).

Test resolution from inside a pod against the cluster DNS service, using the full name svc.namespace.svc.cluster.local. Pods get a search list and ndots:5 in /etc/resolv.conf, so short names are tried with each search domain appended — which is why cross-namespace short names fail and external lookups make extra queries.

A Service only routes to pods whose labels match its selector and that are Ready. Empty endpoints means either the selector matches nothing (label typo, wrong namespace) or the matching pods fail their readiness probe. Compare the Service selector with actual pod labels, then check readiness.

The kubelet turns a 1500m limit into 150ms of CPU per 100ms period, shared by all threads. Forty-eight busy threads spend it in about 3ms and then stall for 97ms, while the average stays low.

Check that the container has a CPU limit, then compare its throttled CFS periods with its total periods: cAdvisor's container_cpu_cfs_throttled_periods_total against container_cpu_cfs_periods_total, or nr_throttled in cpu.stat. A rising ratio confirms throttling.

kubectl port-forward opens a local TCP listener and tunnels it through the API server to the kubelet and into one pod's network namespace. It is TCP-only, single-pod, unbalanced and dies with the connection — a debugging and admin tool, never production access.

Use -l for label selectors (equality and set-based) and --field-selector for a small whitelist of built-in fields such as status.phase and spec.nodeName; both filter server-side. Shape output with -o wide, custom-columns, jsonpath, and --sort-by, which are client-side.

The runtime writes each container's output to /var/log/pods/<namespace><pod><uid>/<container>/<restart>.log, symlinked from /var/log/containers. The kubelet rotates it at containerLogMaxSize (10Mi), keeps containerLogMaxFiles (5), and kubectl logs reads only the current file.

The kubelet cannot assemble the container's configuration, almost always because a referenced ConfigMap, Secret or a specific key inside one does not exist. The container is never created, so there is no exit code and no logs — the detail is in the pod's Events.

SSH to a control-plane node. Check the kubelet, then use crictl ps -a and crictl logs on kube-apiserver and etcd, and inspect /etc/kubernetes/manifests. Check etcd quorum first, because an apiserver restart loop is often an etcd symptom.

kubectl debug node/NODE -it --image=busybox creates a privileged pod pinned to that node in the host network, PID and IPC namespaces, with the node's root filesystem mounted at /host. Run chroot /host to work as if on the node. Delete the pod afterwards.