The shipment-tracking API reads its database password from a Kubernetes Secret synced by the External Secrets Operator: after the password rotates upstream, how do running Pods get the new value without a redeploy?

seniormust knowfreq 58%

answer

three hops, three delays
env is frozen at start
subPath never refreshes
reload or roll
both passwords valid meanwhile

basics

~20 s

The operator updates the Secret within one refreshInterval, but only volume-mounted files change in running Pods, never env vars. Mount files the app re-reads, or trigger a rolling restart, and keep both passwords valid until every Pod has switched.

solid answer

~50 s

Rotation has three hops, and each has its own delay. First, the External Secrets Operator re-reads the provider after at most one refreshInterval (or immediately if you change the ExternalSecret) and updates the Secret in place under the same name. Second, the kubelet propagates that change only into secret volumes mounted without subPath, and only eventually. Env vars from secretKeyRef or envFrom never change in a running container. Third, the process must actually use the new value: either it watches or re-reads the file (and re-establishes its connection pool), or something rolls the Pods, such as kubectl rollout restart or a controller that restarts workloads whose Secret hash changed. None of this needs a new image or a manifest change. Because Pods switch at different moments, the database must accept both passwords during the overlap, and the old one is revoked only after the last Pod has moved.

Rotation has three hops, and each has its own delay. First, the External Secrets Operator re-reads the provider after at most one `refreshInterval` (or immediately if you change the ExternalSecret) and updates the Secret in place under the same name. Second, the kubelet propagates that change only into **secret volumes mounted without `subPath`**, and only eventually. **Env vars from `secretKeyRef` or `envFrom` never change** in a running container. Third, the process must actually use the new value: either it watches or re-reads the file (and re-establishes its connection pool), or something rolls the Pods, such as `kubectl rollout restart` or a controller that restarts workloads whose Secret hash changed. None of this needs a new image or a manifest change. Because Pods switch at different moments, the database must accept **both** passwords during the overlap, and the old one is revoked only after the last Pod has moved.

code

yaml · 29 lines

yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tracking-api
  namespace: shipment-tracking
spec:
  replicas: 9
  selector:
    matchLabels:
      app: tracking-api
  template:
    metadata:
      labels:
        app: tracking-api
    spec:
      containers:
        - name: api
          image: registry.example.com/tracking-api:4.17.2
          env:
            - name: DB_PASSWORD_FILE
              value: /etc/tracking-db/DB_PASSWORD
          volumeMounts:
            - name: db
              mountPath: /etc/tracking-db
              readOnly: true
      volumes:
        - name: db
          secret:
            secretName: tracking-db

go deeper

for a junior

Remember that environment variables are fixed when a container starts, while a mounted Secret volume can change underneath it.

for a middle

Explain the three hops and their delays: the operator's refresh interval, kubelet volume propagation, and whether the process re-reads the value. Mention the subPath exception.

for a senior

Design the rotation end to end: file mounts or automated restarts, a forced sync, alerting on failed syncs, and an overlap window sized from the real delays before revoking.

for a principal

Set a platform standard for how workloads consume rotating credentials, and trade application reload work against restart churn and the overlap the credential owner can grant.

Why "rotation without redeploy" is the real question

A credential rotated in the secret manager is useless until the process holding the old value switches. Interviewers ask this because the change passes through several independent systems, each with its own delay, and a design that forgets one of them either leaks the old credential for days or causes an outage when the old one is revoked.

The scenario: the shipment-tracking API runs 9 replicas in the shipment-tracking namespace. Its database password lives in a secret manager and is synced by an ExternalSecret with refreshInterval: 15m into a Secret named tracking-db.

Hop 1: provider to Secret

The External Secrets Operator re-reads the provider when the refresh interval elapses (default 1h0m0s, here 15 minutes) and updates the same Secret name in place.
To shorten this hop right after a planned rotation, change the ExternalSecret, for example by bumping an annotation. The operator refreshes whenever the ExternalSecret changes.
Worst-case lag for this hop is one interval: up to 15 minutes here, or up to an hour with the default.

Hop 2: Secret to the container's view

How the Pod consumes the Secret decides whether anything changes at all:

Consumption	Updates in a running container?
`env` with `valueFrom.secretKeyRef`	No. Values are resolved once at container start
`envFrom` with `secretRef`	No, for the same reason
`secret` volume, whole directory	Yes, eventually. The kubelet swaps the files atomically
`secret` volume with `subPath`	No. The file is bind-mounted once and never refreshed
Secrets Store CSI Driver volume	Only if the driver runs with rotation enabled

For a whole-directory mount, the delay is the kubelet's sync period (syncFrequency, default one minute) plus the time for its watch-based cache to notice, so expect about a minute or two rather than instant.

Hop 3: container view to the process

A changed file does nothing if the application read it once at startup. There are two honest options:

Hot reload. The application watches the mounted file, or re-reads it on authentication failure, and rebuilds its database connection pool. This is the only zero-restart path, and it needs application code.
Rolling restart. Something replaces the Pods so they start with the new value:
- kubectl rollout restart deployment/tracking-api writes the kubectl.kubernetes.io/restartedAt annotation into the Pod template, and the Deployment rolls within its maxSurge and maxUnavailable budget. No image changes and no manifest edit in Git.
- A reloader-class controller watches Secrets and patches a hash annotation into the Pod template of every workload that references the changed Secret, which automates the same restart.
- With env-var consumption, a restart is the only way.

A rolling restart is not a redeploy in the sense interviewers mean: no new build, no new release, and the same manifests.

The overlap window

Pods switch at different moments, so for a while some use the old password and some the new one. The rotation must therefore follow two-phase logic:

Create the new credential and make it valid alongside the old one.
Publish it to the secret manager and let hops 1 to 3 complete.
Verify that every Pod uses the new value, through application metrics, database session attribution, or the Pod start times after the restart.
Revoke the old credential.

For the tracking API the overlap must cover at least 15 minutes (hop 1), plus about 2 minutes (hop 2), plus the time to roll 9 replicas (hop 3). Revoking the old password at minute 5 is exactly how teams cause the outage they were trying to avoid. The provider's own rotation machinery and dynamic credentials are the secret manager's business. This leaf's concern is the in-cluster path.

Checklist

Prefer file mounts without subPath for anything that rotates.
Size refreshInterval against the overlap window the credential owner will grant.
Automate the restart, or the reload, so rotation needs no human.
Alert when an ExternalSecret's Ready condition is False, because a failing sync silently halts rotation.
Keep old and new credentials valid until the last Pod has switched.

## Why "rotation without redeploy" is the real question A credential rotated in the secret manager is useless until the **process** holding the old value switches. Interviewers ask this because the change passes through several independent systems, each with its own delay, and a design that forgets one of them either leaks the old credential for days or causes an outage when the old one is revoked. The scenario: the shipment-tracking API runs 9 replicas in the `shipment-tracking` namespace. Its database password lives in a secret manager and is synced by an `ExternalSecret` with `refreshInterval: 15m` into a Secret named `tracking-db`. ## Hop 1: provider to Secret - The **External Secrets Operator** re-reads the provider when the refresh interval elapses (default `1h0m0s`, here 15 minutes) and updates the **same** Secret name in place. - To shorten this hop right after a planned rotation, change the ExternalSecret, for example by bumping an annotation. The operator refreshes whenever the ExternalSecret changes. - Worst-case lag for this hop is one interval: up to 15 minutes here, or up to an hour with the default. ## Hop 2: Secret to the container's view How the Pod consumes the Secret decides whether anything changes at all: | Consumption | Updates in a running container? | |---|---| | `env` with `valueFrom.secretKeyRef` | No. Values are resolved once at container start | | `envFrom` with `secretRef` | No, for the same reason | | `secret` volume, whole directory | Yes, eventually. The kubelet swaps the files atomically | | `secret` volume with `subPath` | No. The file is bind-mounted once and never refreshed | | Secrets Store CSI Driver volume | Only if the driver runs with rotation enabled | For a whole-directory mount, the delay is the kubelet's sync period (`syncFrequency`, default one minute) plus the time for its watch-based cache to notice, so expect about a minute or two rather than instant. ## Hop 3: container view to the process A changed file does nothing if the application read it once at startup. There are two honest options: 1. **Hot reload.** The application watches the mounted file, or re-reads it on authentication failure, and rebuilds its database connection pool. This is the only zero-restart path, and it needs application code. 2. **Rolling restart.** Something replaces the Pods so they start with the new value: - `kubectl rollout restart deployment/tracking-api` writes the `kubectl.kubernetes.io/restartedAt` annotation into the Pod template, and the Deployment rolls within its `maxSurge` and `maxUnavailable` budget. No image changes and no manifest edit in Git. - A reloader-class controller watches Secrets and patches a hash annotation into the Pod template of every workload that references the changed Secret, which automates the same restart. - With env-var consumption, a restart is the **only** way. A rolling restart is not a redeploy in the sense interviewers mean: no new build, no new release, and the same manifests. ## The overlap window Pods switch at different moments, so for a while some use the old password and some the new one. The rotation must therefore follow **two-phase** logic: 1. Create the new credential and make it valid **alongside** the old one. 2. Publish it to the secret manager and let hops 1 to 3 complete. 3. Verify that every Pod uses the new value, through application metrics, database session attribution, or the Pod start times after the restart. 4. Revoke the old credential. For the tracking API the overlap must cover at least 15 minutes (hop 1), plus about 2 minutes (hop 2), plus the time to roll 9 replicas (hop 3). Revoking the old password at minute 5 is exactly how teams cause the outage they were trying to avoid. The provider's own rotation machinery and dynamic credentials are the secret manager's business. This leaf's concern is the in-cluster path. ## Checklist - Prefer **file mounts without `subPath`** for anything that rotates. - Size `refreshInterval` against the overlap window the credential owner will grant. - Automate the restart, or the reload, so rotation needs no human. - Alert when an ExternalSecret's `Ready` condition is `False`, because a failing sync silently halts rotation. - Keep old and new credentials valid until the last Pod has switched.

The app reads DB_PASSWORD from secretKeyRef and the team refuses to change code. What is the least-bad setup?
Keep the env var, and automate a rolling restart whenever the Secret's data changes, using a reloader-class controller or a pipeline step that runs `kubectl rollout restart`. Size the overlap window to cover the refresh interval plus a full rollout, and only then revoke the old password.
Why does a Secret mounted with subPath keep the old password even after the kubelet has synced?
With `subPath` the kubelet bind-mounts a single file once at container start. Updates replace the files in the volume's directory through an atomic symlink swap, but the bind mount still points at the old file, so the container never sees the change until it restarts.
How do you prove every replica has switched before revoking the old credential?
Use evidence from the consumer side: all Pods started after the Secret's update time (for the restart path), application metrics that report the credential version in use, or database session attribution showing no logins with the old credential over a full refresh window.

saying these in an interview costs you the question

Updating the Secret updates environment variables in running Pods
The operator restarts Pods when it syncs a new value
A subPath mount picks up Secret changes like a full mount
Revoke the old password as soon as the new one is stored
Rotation needs a new image or a manifest release
Kubelet volume updates are instant

The shipment-tracking API reads its database password from a Kubernetes Secret synced by the External Secrets Operator: after the password rotates upstream, how do running Pods get the new value without a redeploy?

§ answer

basics

solid answer

code

§ go deeper

▾level notes4 levels

▾deep dive~4 min

Why "rotation without redeploy" is the real question

Hop 1: provider to Secret

Hop 2: Secret to the container's view

Hop 3: container view to the process

The overlap window

Checklist

▾common follow-ups3 questions

▾red flags6 flags

Why "rotation without redeploy" is the real question

Hop 1: provider to Secret

Hop 2: Secret to the container's view

Hop 3: container view to the process

The overlap window

Checklist

answer

go deeper