skip to content

When giving a container a Kubernetes Secret, what are the trade-offs between injecting it as environment variables (`envFrom` / `valueFrom.secretKeyRef`) and mounting it as a volume?

level: middleimportance: must knowfreq 58%

answer

  1. env = snapshot at exec; rotation needs restart
  2. volume = tmpfs + kubelet refresh (~1 min)
  3. atomic swap via ..data symlink
  4. subPath NEVER updates
  5. /proc/<pid>/environ, core dumps, child processes

basics

~20 s

Environment variables are captured once at process start, so rotation needs a Pod restart, and they leak easily via /proc, crash dumps and child processes. Volume files are memory-backed and refreshed by the kubelet within about a minute, so apps that re-read the file pick up rotation — but subPath mounts never update.

solid answer

~60 s

**Environment variables** are read into the process at exec time and frozen there. A rotated Secret reaches the container only when the Pod restarts. They also spread easily: `/proc/<pid>/environ`, core dumps, crash reporters, every child process, and any framework that logs its environment on startup. They are convenient — twelve-factor apps expect them — and a missing key fails the Pod fast unless marked `optional: true`. **Volume mounts** project each key as a file on a `tmpfs` (memory-backed) mount, so nothing hits node disk. The kubelet refreshes the contents when the Secret changes — typically within the kubelet sync period plus its cache TTL, so on the order of a minute — and does so atomically by swapping a symlink to a new `..data` directory, so a reader never sees a half-written file. An application that re-reads the file gets rotation with no restart. You can also control `defaultMode` and `fsGroup`. The critical exception: a mount using `subPath` is **not** updated, ever. Default to volumes for anything rotating (TLS material, DB passwords with rotation); env vars are acceptable for static values in simple apps.

code

yaml · 26 lines
yaml
apiVersion: v1
kind: Pod
metadata:
  name: api
spec:
  containers:
    - name: app
      image: registry.example.com/api:2.1.0
      env:
        - name: DB_PASSWORD          # snapshot at start; rotation needs a restart
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: password
      volumeMounts:
        - name: tls                  # updated in place by the kubelet
          mountPath: /etc/tls
          readOnly: true
        # - name: tls
        #   mountPath: /etc/tls/tls.crt
        #   subPath: tls.crt        # NEVER updated - avoid for rotating material
  volumes:
    - name: tls
      secret:
        secretName: api-tls
        defaultMode: 0400

go deeper

for a junior

Know the headline: env vars are fixed once the container starts, mounted files can be updated by the kubelet, and both are readable inside the Pod.

for a middle

Explain the refresh mechanism (tmpfs, kubelet sync, atomic ..data symlink swap), the subPath exception, and the concrete env-var leak paths.

for a senior

Tie the choice to rotation strategy — whether a credential change is a file refresh or a rollout — and cover failure modes, optional: true, file permissions and the propagation window during rotation.

for a principal

Set the platform default (volumes plus re-read-on-use for anything rotating), define the app contract for reloading, and weigh it against short-lived credentials that make static Secrets unnecessary in the first place.

## The two injection paths A Secret reaches a container in one of two ways. **As environment variables**, via `envFrom.secretRef` (all keys, names become variable names) or `env[].valueFrom.secretKeyRef` (one key, explicitly named). The kubelet reads the Secret at container-start time and passes the values in the process environment when it execs the entrypoint. **As a volume**, via `volumes[].secret` and a `volumeMounts` entry. The kubelet materialises each key as a file inside the mount directory. ## Freshness: the decisive difference Process environments are immutable after exec on Linux from the outside — nothing can change a running process's environment for it. So an env-injected Secret is a snapshot taken when that container started. Rotate the Secret and the running container keeps the old value indefinitely; it only picks up the new one after a restart, which means a rotation is a rollout. Volumes are live. The kubelet keeps the mounted content in sync with the API object. Propagation is not instant: it is bounded by the kubelet sync loop plus the object cache TTL, so plan for roughly a minute rather than milliseconds. The update is **atomic**, which matters more than people expect. The kubelet writes a new timestamped directory containing all keys, then atomically swaps the `..data` symlink to point at it. Files visible in the mount are symlinks through `..data`. Consequently a reader either sees the entire old set or the entire new set — never a TLS key from the new pair with a certificate from the old. This is why cert-manager-style rotation works safely through volumes. **`subPath` breaks all of this.** Mounting a single key with `subPath: tls.crt` bind-mounts that one file into the container, bypassing the symlink indirection. The kubelet cannot update it, so the content is fixed for the Pod's lifetime. This is the number-one surprise in production: "we mounted the cert but renewal never reached the Pod". If you need one file at a specific path and still want updates, mount the whole volume elsewhere and symlink, or use `items` to project only the keys you want into a normal (non-subPath) mount. ## Exposure surface Environment variables are broad by nature: - readable at `/proc/<pid>/environ` by anything in the same Pod's PID namespace or by root on the node; - inherited by every child process, including shells and helper binaries; - captured by core dumps and by many crash/APM reporters that snapshot the environment; - printed by frameworks and debug endpoints that dump configuration on startup; - visible in the container runtime's inspect output on the node. Volume files are narrower: one path, permissions you control with `defaultMode` (e.g. `0400`) and `fsGroup`, memory-backed so never written to node disk, and no automatic inheritance into child processes. They are still readable by anyone who can `exec` into the Pod — neither option defends against that. One small point in favour of env vars: `kubectl describe pod` shows a `secretKeyRef` as a *reference*, not a value, so the manifest itself does not leak. The leak paths are inside the container and on the node, not in the API object. ## Failure behaviour With `secretKeyRef`, a missing Secret or missing key stops the container from starting — the Pod sits in `CreateContainerConfigError` — unless you set `optional: true`. With a volume, a missing Secret blocks the mount and the Pod stays in `ContainerCreating` until the Secret appears, again unless `optional: true`. Fast-fail on a missing credential is usually what you want; the `optional` flag is for genuinely optional configuration. ## Practical guidance - Rotating material — TLS certificates and keys, database credentials under automated rotation, tokens with a lifetime — should be **volumes**, and the application should re-read on use or watch the file. Deliver a rotation without a restart, or accept that rotation equals a rollout. - Static credentials in a simple twelve-factor app, where the operational cost of a restart per change is nil, are fine as **env vars**. Prefer explicit `secretKeyRef` over `envFrom` so you can see exactly which values the container gets and avoid accidentally injecting new keys when someone adds them to the Secret. - Whatever you choose, ensure the app never logs the value and that crash handlers scrub the environment. ## A third option worth naming A `projected` volume can combine several Secrets, ConfigMaps, a Downward API entry and a short-lived service-account token into one directory with explicit paths. It is the tidy choice when a container needs several pieces of material in one place, and it keeps the atomic-update behaviour of ordinary Secret volumes.

  • How quickly does a rotated Secret reach a mounted volume, and what governs the delay?
    On the order of a minute, not instantly. The bound is the kubelet's sync period plus the TTL of its object cache for ConfigMaps and Secrets, so the application must tolerate a window where old and new material coexist across the fleet. If you need immediate cutover, do a rollout instead and treat the volume refresh as best-effort.
  • Why does mounting a Secret key with `subPath` stop it from updating?
    A `subPath` mount bind-mounts that single file into the container, bypassing the `..data` symlink indirection the kubelet uses to swap contents atomically. Since the kubelet updates by repointing that symlink, a bind-mounted file can never be refreshed and stays fixed for the Pod's lifetime. Mount the directory normally, or use `items` to project just the keys you need.

An environment variable is a value photocopied onto the process at birth; a volume file is a noticeboard the kubelet keeps repinning — but pin one sheet directly to the wall with subPath and nobody can replace it.

saying these in an interview costs you the question

  • Believing environment variables update when the Secret changes.
  • Using `subPath` for TLS material and expecting certificate renewal to reach the Pod.
  • Assuming volume updates are instantaneous rather than bounded by the kubelet sync period.
  • Claiming volumes are safe from anyone who can `exec` into the Pod — they are not.
  • Thinking `envFrom.secretRef` is safer than `secretKeyRef`; it is broader, injecting every key including ones added later.

context

resolving links…