How do you run a command or open an interactive shell inside a container of a running Kubernetes pod with kubectl, and what are the limits of that approach?
answer
- exec = new process in a live container
- API server proxies to kubelet, not you to node
- pods/exec subresource RBAC
- needs the binary in the image
- changes die with the container
basics
~20 sUse kubectl exec -it POD -c CONTAINER -- sh. It starts an extra process inside an already-running container, streamed through the API server and the kubelet. The binary must exist in the image and the container must be running.
solid answer
~50 s`kubectl exec -it mypod -c app -- /bin/sh` opens a shell; without `-it` it runs one command and prints the output, e.g. `kubectl exec mypod -- env`. Everything after `--` is the command. Mechanically kubectl calls the pod's `exec` subresource; the API server proxies a streamed connection to the kubelet on that node, which asks the container runtime to start a process in the container's namespaces. That needs `create` on `pods/exec` — read access to pods is not enough. Limits: the binary must exist in the image, so distroless/scratch images have no shell; the container must be running, so a crash-looping container is unreachable this way; anything you install lives in the writable layer and disappears on restart; `-c` is needed for multi-container pods; your process shares the container's CPU and memory limits, so a heavy tool can get the container OOM-killed.
code
bash · 4 lineskubectl exec -n prod mypod -- env
kubectl exec -n prod mypod -c app -- cat /etc/app/config.yaml
kubectl exec -it -n prod mypod -c app -- /bin/sh
kubectl exec -n prod mypod -c app -- curl -s localhost:8080/healthzgo deeper
Recall the command shape, the meaning of -it, -c and --, and that the image must contain the binary you ask for.
Explain the API-server-to-kubelet-to-runtime path, the pods/exec subresource, and why distroless images and crash-looping containers defeat exec.
Frame exec as an audited, privileged mutation: what it can leak, when to prefer ephemeral containers, and why fixes made inside a live container are a defect.
Discuss whether interactive access should exist in production at all, and what observability and break-glass workflow replaces it.
## What the command does `kubectl exec` starts an extra process **inside an existing, running container**. It does not restart anything, create a pod, or change the pod spec. - One-shot: `kubectl exec mypod -- ls /app` runs, prints stdout/stderr, exits with the command's exit code. - Interactive: `kubectl exec -it mypod -- /bin/sh`. `-i` keeps stdin open, `-t` allocates a TTY so line editing and Ctrl-C work. The `--` separator matters: flags before it are kubectl's, everything after belongs to the command in the container. `kubectl exec mypod -- ps -ef` works; leaving out `--` makes kubectl try to parse `-ef`. ## The path a request takes Your client does not talk to the node. kubectl POSTs to the pod's `exec` subresource on the API server; the API server opens a streamed connection to the kubelet on the node hosting the pod; the kubelet asks the container runtime (containerd, CRI-O) to spawn the process in that container's namespaces and cgroup. Consequences: the connection is authenticated and authorized by your kubeconfig identity, it is auditable server-side, and it works without any network route from you to the node. Recent Kubernetes versions carry these streams over WebSockets rather than the older SPDY protocol, which matters only if an intermediate proxy sits between you and the API server. ## Choosing the container A pod can hold several containers. Without `-c`, kubectl picks the container named by the `kubectl.kubernetes.io/default-container` annotation, or the first one, and prints a warning. Always pass `-c` when debugging multi-container pods (app plus sidecar). Init containers cannot be exec'd into — by the time the pod is running they have already finished. ## When exec fails - **No shell in the image.** Distroless and `FROM scratch` images ship no `/bin/sh`, so exec returns a not-found error. This is the main reason ephemeral debug containers exist. - **Container not running.** A container in CrashLoopBackOff spends most of its time terminated; exec returns an error that the container is not running. Read logs with `--previous` instead, or start a copy of the pod with an overridden command. - **RBAC.** `pods/exec` is a separate subresource; a role granting `get`/`list` on pods does not include it. In many clusters it is deliberately withheld in production. - **Admission policy.** Some clusters block exec on certain namespaces with a validating webhook. ## Discipline An exec session is a mutation you cannot see in Git. Files you edit, packages you install, and processes you kill exist only in that container's writable layer and are lost the moment it restarts — which makes exec fine for *reading* state (config on disk, `ps`, `env`, `curl localhost`) and a trap for *fixing* things, because the fix silently disappears and the next replica never has it. Prefer capturing evidence and changing the manifest. Related file transfer uses the same channel: `kubectl cp` is exec plus `tar`.
- The image is distroless and has no shell. How do you get a debugging environment for that pod?Attach an ephemeral container with `kubectl debug -it mypod --image=busybox --target=app`. It runs a tool-bearing image in the same pod without restarting it, and `--target` shares the process namespace of the named container so you can see its processes and reach its filesystem through /proc. If the pod is not running at all, use `kubectl debug --copy-to` to create a modified copy of the pod instead.
- Why do many teams deny pods/exec in production, and what do they lose?Exec grants arbitrary code execution as the container's user with its service-account token and mounted secrets, and lets someone mutate a running replica invisibly. Denying it removes that risk but also removes ad-hoc inspection, so teams compensate with good logs, metrics, admin endpoints, and a break-glass role that grants pods/exec temporarily with audit logging enabled.
Like opening a second terminal on a machine that is already booted — you can look around and run tools that are installed, but you cannot reboot it into a rescue image, and nothing you type is saved to the machine's build recipe.
saying these in an interview costs you the question
- Saying kubectl exec restarts or recreates the container
- Believing kubectl connects directly to the node or over SSH
- Expecting a package installed during an exec session to survive a restart
- Trying to exec into a CrashLoopBackOff container instead of reading previous logs
- Assuming read access to pods implies permission to exec