What's the practical difference between storing a database password as a Kubernetes Secret (base64-encoded, mounted into a pod) versus fetching it at runtime from HashiCorp Vault using dynamic, short-lived credentials? What operational problem does each approach create?

## How a Kubernetes Secret works A **Kubernetes Secret** is an API object storing key-value data that's base64-encoded in etcd by default — base64 is an encoding, not encryption, so anyone with etcd access or the RBAC permission to `get secrets` in that namespace can trivially decode it. Clusters can optionally enable encryption-at-rest for etcd via a KMS-backed envelope key, but that's a cluster-admin opt-in, not the default. A pod consumes a Secret either - as environment variables injected at container start, or - as files mounted into a volume. Either way, the value is **static** — whatever was last written to the object — and it never expires on its own. ## How Vault's dynamic secrets work **HashiCorp Vault's database secrets engine** works differently: it stores only a privileged "root" credential to the database and generates a unique, scoped credential on each request from an authenticated client, with a **TTL** (say one hour) and a **lease** Vault tracks centrally. The requesting service authenticates to Vault, commonly via Kubernetes auth, where Vault validates the pod's service-account JWT against the Kubernetes API, then either 1. calls the Vault API directly, or 2. more commonly, relies on a **Vault Agent** sidecar to fetch and continuously renew the credential. Vault can also revoke leases centrally, cutting off access immediately without touching the underlying database's user table by hand. ## What each one solves Kubernetes Secrets solve the basic problem of keeping credentials out of container images and ordinary ConfigMaps, with tighter RBAC on the distinct `secrets` resource type — already a meaningful step up from baking credentials into an image. Vault's dynamic secrets solve a further problem: a long-lived static credential has a large **blast radius** if leaked, since it stays valid until someone notices and manually rotates it, which could take days; a one-hour-TTL dynamic credential bounds that exposure window automatically, and because each credential is uniquely issued, a compromised instance's leaked credential can be traced and revoked without affecting every other instance that would otherwise share one static Secret. | Kubernetes Secret | Vault dynamic secret | |---|---| | base64-encoded in etcd, with encryption-at-rest an opt-in | a unique, scoped credential on each request from an authenticated client | | static — whatever was last written to the object | a TTL, say one hour, and a lease Vault tracks centrally | | rotate by hand, then restart every pod that consumed it as an environment variable | revoke leases centrally | | zero extra infrastructure | another distributed system that must run in HA | ## The cost The cost is real operational weight. Vault is another distributed system that must - run in HA, - be secured, - be unsealed (Vault starts sealed and needs an unseal mechanism, typically auto-unseal via a cloud KMS), - be backed up, - and be monitored, and it adds a new hard dependency at credential-renewal time that a plain Secret mount never has. Application code either needs **Vault-awareness** — SDK calls and lease-renewal logic — or a **sidecar pattern** that adds a second container per pod, more resource overhead, and another independently-failing moving part. Kubernetes Secrets, by contrast, are simple and "just work" with zero extra infrastructure, but leave you with static, long-lived, weakly-protected-by-default credentials that are hard to rotate and audit at scale. ## Failure modes in production - **In production, static K8s Secrets commonly fail on rotation friction.** Rotating a DB password means updating the Secret, and every pod that consumed it as an environment variable needs a restart to pick up the new value — volume-mounted Secrets update on a sync delay, but env vars never do — so teams either avoid rotation entirely, leaving stale credentials valid indefinitely, or build custom restart orchestration around it. - **Vault dynamic secrets fail differently.** A lease can expire faster than the app renews it under load, GC pauses, or clock skew, causing a DB connection to be suddenly rejected mid-run if renewal logic isn't robust, and cascading failures can occur if Vault itself becomes unreachable and no instance can obtain a fresh lease as its current one expires. ## A common real-world pattern A common real-world pattern is Vault deployed in HA behind Kubernetes auth, with each pod's service-account token exchanged for a Vault token, and a Vault Agent sidecar continuously renewing a database credential written to a shared in-pod volume the app reads — roughly HashiCorp's own Vault-on-Kubernetes reference architecture, adopted by regulated industries like fintechs for PCI-scoped database access, precisely because auditors want proof that database credentials expire and are individually traceable per workload rather than shared static passwords.