How do you read container logs with kubectl, including output from a container instance that has already exited, and why might kubectl logs return nothing at all?
answer
- stdout/stderr only, files on the node
- --previous = last terminated instance, one kept
- --since / --tail bound the volume
- -c for sidecars, -l across replicas
- deleted pod = logs gone
basics
~20 skubectl logs POD -c CONTAINER shows the current instance; --previous shows the last terminated one. Add -f to follow, --since and --tail to bound output. Nothing appears if the app logs to a file instead of stdout/stderr, the pod never started, or the logs rotated.
solid answer
~50 s`kubectl logs mypod -c app` prints the current container instance's stdout and stderr. Useful flags: `-f` to follow, `--previous`/`-p` for the instance that died (essential for crash loops, because the running one has just started), `--tail=200`, `--since=15m`, `--timestamps`, `--all-containers=true`, and `-l app=api --prefix` to read across replicas. Logs are not stored in the API server: the container runtime writes stdout and stderr to files on the node and the kubelet serves them, so kubectl proxies to the node. That explains the empty cases. If the app writes to a file inside the container, nothing is captured. If the pod is Pending or the image never pulled, no container ever ran. `--previous` fails when the container has not restarted, or when it is a **new pod** — a recreated pod has no history of its predecessor. Rotation (default around 10Mi per file) drops older output, and deleting the pod removes the logs entirely.
code
bash · 4 lineskubectl -n prod logs api-7d9f-abcde -c api --tail=200 --timestamps
kubectl -n prod logs api-7d9f-abcde -c api --previous
kubectl -n prod logs -f deploy/api --since=5m
kubectl -n prod logs -l app=api --all-containers=true --prefix --tail=50go deeper
Know the command, -c, -f, and that --previous shows the container instance that died.
Explain where logs physically live, rotation, and the concrete reasons output can be empty.
Bound queries with --since and --tail, read across replicas with selectors, and preserve snapshots before mutating anything; know when kubectl stops being enough and shipped logs are required.
Set the standard: stdout-only structured logging, retention through a log pipeline, and the expectation that node-local logs are best-effort evidence.
## Where logs come from A container's stdout and stderr are captured by the container runtime and written to files on the node, typically under `/var/log/pods/<ns>_<pod>_<uid>/<container>/`. The kubelet serves them over its API; `kubectl logs` calls the pod's `log` subresource on the API server, which proxies to that kubelet. Three consequences follow immediately: only stdout and stderr are visible, logs live on the node and die with the pod, and there is no search, no retention beyond rotation, and no cross-pod aggregation — that is what a log pipeline is for. ## The flags worth memorising - `-c <container>` — required for multi-container pods unless the default-container annotation is set. Init containers are addressed the same way. - `--previous` / `-p` — the **last terminated instance** of that container. Kubernetes keeps one previous instance's log; this is how you see why a crash-looping container died, since the currently running instance has only just started. - `-f` — follow, like `tail -f`. Combine with `--tail` so you do not replay hours of history first. - `--since=10m` or `--since-time=<RFC3339>` — bound by time; `--tail=N` bounds by lines. Without either, kubectl streams the whole file, which on a chatty service is slow and can be large. - `--timestamps` — kubelet-recorded timestamps, valuable when correlating with Events whose clocks differ from the app's own log formatting. - `--all-containers=true` — every container in the pod at once; `--prefix` labels each line with its source. - `-l <selector>` with `--max-log-requests` — read many pods of a Deployment at once, which is how you find the one bad replica. ## Why logs can be empty 1. **The app does not log to stdout or stderr.** A framework configured to write `/var/log/app.log` inside the container produces nothing for kubectl. The fix is configuration, not tooling: containerised apps log to stdout. 2. **No container ever ran.** A `Pending` pod, a failing image pull, or a failing init container means there is no log for the main container; `kubectl describe` and the pod's Events carry the reason instead. 3. **Wrong container.** In a pod with a sidecar, the default container may be the quiet one. 4. **Buffered output.** Some runtimes buffer stdout when it is not a TTY, so a Python or Node process can appear silent for a long time. Unbuffered output (for example `PYTHONUNBUFFERED=1`) is the fix. 5. **Rotation.** The kubelet rotates container logs (commonly 10Mi per file, a handful of files kept), so a verbose service loses old lines within minutes. 6. **The pod is gone.** Deleting the pod — or a node failure, or eviction — removes the log files. A pod recreated by a Deployment is a different pod with no access to its predecessor's output, which is exactly why `--previous` errors in that case rather than showing the history people expect. ## What --previous covers precisely `--previous` works when the *same pod* has a container that terminated and was restarted in place, which is the normal CrashLoopBackOff shape. It returns an error if the container has never restarted, and only one prior instance is retained — a container that has restarted 40 times gives you attempt 39, not attempt 1. If you need the full history of a repeatedly failing workload, you need shipped logs. ## Practical habit During triage, take a bounded snapshot rather than a live tail: `kubectl logs mypod -c app --previous --tail=500 --timestamps > previous.log`. It preserves evidence that a pod deletion would otherwise destroy, and it is easy to attach to a ticket. Then use `-f` for watching a fix take effect.
- A pod has restarted 40 times and you need the very first failure. Can kubectl give it to you?No. Kubernetes retains only the current and one previous container instance's logs for a pod, so kubectl logs --previous shows the most recent failed attempt, not the first. Earlier attempts, and any output from pods that were replaced entirely, are available only if logs were shipped off the node to a log store.
- An application writes its logs to /var/log/app.log inside the container. What do you change?Reconfigure the application to log to stdout and stderr so the runtime captures the stream and both kubectl logs and any log shipper can see it. Writing to a file inside the container also fills the writable layer and is lost on restart. If the framework truly cannot be changed, a sidecar tailing a shared emptyDir volume to stdout is the usual workaround.
saying these in an interview costs you the question
- Believing logs are stored in the API server or etcd
- Expecting --previous to show the full restart history rather than one prior instance
- Forgetting -c on multi-container pods and concluding the app is silent
- Assuming a recreated pod's logs include the deleted pod's output
- Not knowing that only stdout and stderr are captured