Why do hardened Kubernetes deployments prefer mounting a Secret as a file through a volume rather than injecting it into the container as an environment variable with secretKeyRef, and how does that choice affect what happens when the Secret's value is later rotated?

## Two delivery mechanisms A Secret reaches a container in one of two shapes. Either the Pod spec references keys through `env[].valueFrom.secretKeyRef` (or pulls a whole Secret with `envFrom.secretRef`), in which case the kubelet resolves the values at container creation and passes them in the process environment; or the Pod declares a `secret` volume (or a `projected` volume containing one) and the kubelet materialises each key as a file under the mount path, on a tmpfs. ## Why the environment is a poor place for a credential The environment block is part of the process image, and the operating system offers several ways to read it: - `/proc/<pid>/environ` exposes it to any process in the container that can see that PID, which includes sidecars sharing a process namespace and anything running as root in the container. - Every child process inherits the environment by default, so a shell-out, a subprocess, or a plugin all receive the credential whether or not they need it. - Runtime introspection surfaces dump it: crash handlers and error-reporting SDKs commonly attach the environment to reports sent to an external SaaS; framework debug endpoints (`/actuator/env`, `/debug/vars`-style handlers, phpinfo pages) print it; a stack trace including the environment ends up in the log pipeline, which usually has much broader read access than the `secrets` RBAC resource. - `kubectl describe pod` prints literal `env` values in full. It only redacts to a reference when the value comes from `secretKeyRef`, so a copy-pasted literal is fully visible to anyone with `get pods`. A file has none of these properties. It is read by the one code path that opens it, its exposure is bounded by file mode and the container's user, and it is not implicitly propagated to children. ## Rotation behaviour: the difference that surprises people Environment variables are resolved exactly once, at container start. Updating the Secret afterwards has no effect on running containers: the process environment is immutable from the outside. The only way to pick up a new value is to recreate the Pod, which teams usually automate by stamping a checksum of the Secret into a Pod-template annotation so a change rolls the Deployment. Volume-mounted Secrets are different. The kubelet keeps mounted Secrets fresh; with the default watch-based change-detection strategy the file content is replaced within roughly the sync period, on the order of a minute. The update is atomic: the kubelet writes a new timestamped directory and swings the `..data` symlink, so a reader never sees a half-written file. The application still has to notice: either re-read the file per use, watch it with inotify, or reload on a signal. Nothing about a mounted Secret makes a long-lived database connection pick up a new password by itself. One sharp edge: a volume mounted with `subPath` is resolved at mount time and is *never* refreshed. Using `subPath` to drop a single file next to existing files in a directory silently converts a rotating Secret back into a frozen one. ## When environment variables are still acceptable They are not forbidden. Twelve-factor-style apps and many third-party images only accept configuration through the environment, and rewriting them is not always worth it. The pragmatic rule: if a value is short-lived, low-value, or you were going to restart on rotation anyway, `secretKeyRef` is fine. For long-lived, high-value credentials, and especially for anything an external secret manager rotates on a schedule, mount a file. If an image insists on an environment variable, an entrypoint wrapper that reads the file and execs the real process keeps the value out of the manifest and out of the parent environment for less time. ## What to say in an interview Lead with the two concrete failure modes: environment capture (`/proc/<pid>/environ`, child processes, crash reporters, debug endpoints) and rotation freeze. Then mention the kubelet's atomic in-place refresh, the `subPath` exception, and the fact that the application must still re-read the file for a new value to take effect.