How would you make Kubernetes Secret data unreadable to someone who obtains a raw copy of the etcd data files or a cluster backup, and what does a KMS provider add over an encryption key configured locally on the control plane?

## The threat being addressed Encryption at rest defends against an attacker who gets the bytes without going through the API server: a stolen or decommissioned control-plane disk, an etcd snapshot sitting in an object-storage bucket, a backup restored into a lab, or direct etcd client access. It does not defend against an attacker who can call the API server with sufficient RBAC, because the API server necessarily returns plaintext. RBAC and encryption at rest are complementary, not alternatives. ## Configuring it The API server takes `--encryption-provider-config=/etc/kubernetes/enc.yaml`, an `EncryptionConfiguration` object listing resources and, for each, an ordered list of providers. The available providers are `identity` (no encryption), `secretbox` (XSalsa20-Poly1305), `aesgcm`, `aescbc`, and `kms`. `aesgcm` requires strict key rotation discipline because of nonce reuse risk, so `aescbc` or `secretbox` are the usual local choices and `kms` is the recommended one. Two rules govern the list. The **first** provider is used to encrypt every write. **All** providers are tried, in order, when decrypting, which is what makes rotation possible. Consequently, `identity` must be last if it appears at all, and putting `identity` first silently disables encryption while looking configured. Because encryption happens on write, enabling it changes nothing about objects already in etcd. You must rewrite them, typically with `kubectl get secrets --all-namespaces -o json | kubectl replace -f -`. The same command is how you complete a key rotation after prepending a new key. Verify by reading a key straight out of etcd with `etcdctl` and confirming the value starts with a provider prefix such as `k8s:enc:kms:v2:` rather than showing plaintext. Also remember there are usually multiple API server replicas: every one of them needs the same configuration file before you start rewriting, or writes served by an unconfigured replica stay in plaintext and reads of newly encrypted objects fail. ## What a local key does and does not buy With `aescbc` or `secretbox`, the key material sits base64-encoded in a YAML file on each control-plane node, readable by root. That is genuinely useful: an etcd backup leaked to a bucket, a snapshot copied to a laptop, or a disposed disk becomes useless on its own. But the key travels with the machine, so anyone who compromises a control-plane node, or who obtains both the backup and a copy of the config, gets everything. There is no audit trail of key use and no way to revoke. ## What a KMS provider adds A `kms` provider moves the root of trust out of the cluster. In KMS v2 (GA in Kubernetes 1.29; v1 is deprecated) the API server generates a local key-encryption key, has the external KMS wrap it, and caches the result; individual objects are encrypted with data keys under that hierarchy, so the number of calls to the external KMS stays small while the key that matters never exists in cleartext on the node in a durable form. The provider runs as a socket-based plugin the API server talks to over a UNIX domain socket. Concretely this gives you: key material held in a managed KMS or HSM with its own access policy; every use of the root key logged in the cloud provider's audit trail; revocation as a real operation, since disabling the KMS key makes existing ciphertext undecryptable; and rotation performed in the KMS, with KMS v2 able to pick up a new key version without an API server restart (`status.key_id` changes and the API server re-wraps). The tradeoff is a hard availability dependency. If the KMS is unreachable, the API server cannot decrypt Secrets, so Secret reads fail and, in practice, so do Pod starts that need them. You size for that: regional KMS endpoints, caching, and a tested plan for a KMS outage, plus a documented recovery path since an unrecoverable key loss means unrecoverable Secrets. ## Adjacent hygiene Encryption at rest is one control among several for the same threat. etcd should also be on encrypted storage, reachable only over mutual TLS from the API servers, and never exposed on a routable network. Backups inherit whatever protection etcd had at write time, so an encrypted-at-rest cluster produces backups that are safe to store but still worth encrypting again in transit and in the bucket. And the audit policy should keep Secret request and response bodies out of the audit log, since audit sinks are usually far less protected than etcd. ## What to say Name the flag and the file, state the write-first/read-all provider ordering, mention that existing objects need a rewrite, then contrast a local key (defends the backup, not the node) with a KMS provider (external root of trust, auditable and revocable, at the price of an availability dependency).