Explain why secrets (database passwords, API keys, encryption keys) are usually not stored the same way as ordinary configuration values like timeouts, even when both live in the same external configuration store, and what an integration with a dedicated secrets manager such as HashiCorp Vault or AWS Secrets Manager adds beyond plain external config.

## Blast radius is the whole distinction The distinction between ordinary configuration and secrets comes down to **blast radius**: what happens if the value leaks or is read by someone who shouldn't see it. | Value | What its exposure costs | |---|---| | A connection-pool size or an HTTP timeout leaking to an unauthorized party | essentially harmless - at worst it reveals mildly useful operational detail | | A database password, an API key, or an encryption key | gives an attacker direct, standing access to a real system: they can read or write production data, impersonate the service to a third-party API, or decrypt data that was supposed to be protected | Because the consequences of exposure are categorically different, the storage, access-control, and lifecycle requirements need to be categorically different too, even though both 'ordinary config' and 'secrets' are conceptually just key-value pairs an application reads at startup. ## Why a plain config store is dangerous for secrets A plain external configuration store is typically optimized for simplicity, fast reads, and easy human editing - config is often stored as plaintext YAML/JSON in a Git repo or a KV store with namespace-level access control (e.g., 'anyone on the platform team can read/write the prod namespace'). That model is dangerous for secrets for several concrete reasons. 1. **First, plaintext-at-rest** means anyone with read access to the underlying storage (a Git repo clone, a database backup, a KV store snapshot) has the secret in the clear, with no additional barrier. 2. **Second, namespace-level ACLs are too coarse:** 'anyone who can read prod config' is a much larger set of people than 'anyone who needs the payments-service database password specifically,' and secrets need per-secret, often per-identity granularity. 3. **Third, ordinary config stores rarely audit reads** (only writes/changes), but for secrets, knowing who read a credential and when is essential for incident response - if a secret might be compromised, you need to know exactly which services or people accessed it to scope the blast radius. 4. **Fourth, secrets need to be rotated periodically** as a security hygiene practice, and rotating a value that's hardcoded into a flat config file means coordinating a synchronized update across every consumer, whereas a proper secrets manager can generate new credentials, keep the old ones valid for a brief overlap window, and let each consumer pick up the new value on its own schedule. ## What a dedicated secrets manager adds A dedicated secrets manager like **HashiCorp Vault** or **AWS Secrets Manager** is purpose-built to close these gaps. It provides: - **Encryption at rest by default**, typically using envelope encryption (the secret is encrypted with a data key, which is itself encrypted with a master key managed by a hardware security module or a cloud KMS), so even someone with raw storage access sees only ciphertext. - **Identity-based access policies** rather than flat namespaces - a specific service's IAM role, Kubernetes service account, or Vault AppRole is granted read access to exactly the secrets it needs, following least-privilege, and every access attempt (successful or denied) is logged for audit. **Advanced secrets managers go further with dynamic secrets:** instead of a long-lived static database password stored anywhere at all, the secrets manager generates a short-lived, unique database credential on demand when a service requests one, valid for a bounded time window and automatically revoked afterward - meaning there's no long-lived secret sitting in storage for an attacker to steal in the first place. **Automatic rotation** extends this to credentials that must remain relatively static (like a third-party API key that can't be dynamically generated): the secrets manager rotates the underlying credential on a schedule and coordinates the update so consuming services pick up the new value, often with a brief dual-validity window so in-flight requests using the old credential don't fail mid-rotation. ## How the two blend in practice Many real systems blend the two: application code still reads secrets through the same configuration-loading code path used for ordinary settings, for a consistent developer experience, but under the hood the config client resolves secret-shaped keys by calling out to Vault or AWS Secrets Manager (often via a sidecar like the **Vault Agent**, or a Kubernetes **CSI Secrets Store driver** that mounts secrets as files) rather than reading them from the plain config namespace. The failure mode to watch for is a team treating this integration as optional convenience and, under time pressure, dropping an API key into the plain config store 'just for now' - which quietly reintroduces plaintext, coarse-ACL, unaudited, unrotatable secret storage exactly where the team thought they'd designed it out. A well-known incident pattern is a secret committed to a config repo's Git history: even after the value is later moved to a proper secrets manager and 'removed,' it remains recoverable from the repo's history unless that history is explicitly purged and the credential is rotated - which is exactly why secrets should never be typed into a plain config file in the first place, not even temporarily.