How do you ensure Kubernetes Secret values are not written to etcd in plaintext, and what happens to Secrets that already exist when you enable that?

## What the feature is By default, `kube-apiserver` serialises a Secret and hands the bytes to etcd unchanged. Anyone with the etcd data files, an etcd snapshot, a control-plane volume snapshot, or direct etcd client access reads every credential. Encryption at rest closes that specific hole by having the API server encrypt the value before it is stored and decrypt it on read. etcd itself knows nothing about it. ## The configuration file ```yaml apiVersion: apiserver.config.k8s.io/v1 kind: EncryptionConfiguration resources: - resources: ["secrets"] providers: - kms: apiVersion: v2 name: cloudkms endpoint: unix:///var/run/kmsplugin/socket.sock - identity: {} ``` Passed to the API server as `--encryption-provider-config=/etc/kubernetes/enc/enc.yaml`. Since Kubernetes 1.30 `--encryption-provider-config-automatic-reload` (GA) lets the file be re-read without restarting the process, which makes rotation far less painful. **Provider ordering is the whole game.** The first entry encrypts writes; every entry is available for decryption, matched by the prefix stamped into the stored blob (`k8s:enc:kms:v2:...`, `k8s:enc:aescbc:v1:...`, or none for identity). Therefore: - `[kms, identity]` → new writes encrypted, old plaintext still readable. This is what you deploy first. - `[identity, kms]` → writes plaintext, existing encrypted data still readable. This is the *decommission* configuration, used before removing encryption. Getting these backwards silently leaves you unencrypted, which is why the ordering question is the one interviewers ask. ## Provider options - **`identity`** — no encryption. Legitimate only as a fallback entry or during migration. - **`secretbox`** / **`aesgcm`** / **`aescbc`** — local providers with the key material written in the config file on the control-plane node. `aesgcm` requires careful key rotation (nonce reuse risk) and the docs steer you to `secretbox` or KMS. All of them have the same structural weakness: the key sits on the same machine as etcd, so they protect stolen backups and detached disks, not a compromised control-plane node. - **`kms` v2** (GA since 1.29; v1 deprecated and removed in later releases) — envelope encryption. The API server generates a data-encryption key, has the external KMS wrap it with a key-encryption key that never leaves the KMS/HSM, and caches the DEK. v2 fixed v1's per-object KMS call storm by using a single DEK per API server with a key ID for rotation detection, so it performs well and rotates cleanly. This is the answer for anything with a compliance requirement. ## Rolling it out 1. Write the config to **every** control-plane node and restart/reload every API server. Until all of them have it, an instance without the config cannot decrypt what the others wrote — a partial rollout produces intermittent decryption errors. 2. Verify a new Secret is actually encrypted by reading the raw key from etcd: `ETCDCTL_API=3 etcdctl get /registry/secrets/default/probe | hexdump -C` — you should see the `k8s:enc:` prefix and no readable value. 3. **Rewrite existing objects.** Encryption happens on write; every Secret created before the change is still plaintext in etcd. The classic no-op rewrite is `kubectl get secrets --all-namespaces -o json | kubectl replace -f -`. Modern clusters can instead use the StorageVersionMigration API/storage-version-migrator, which handles large clusters more gracefully and is resumable. Either way, watch for immutable Secrets — `replace` on those fails, and they must be deleted and recreated or migrated by a tool that rewrites without changing data. 4. Rotate keys by adding the new key as the **second** entry everywhere, rolling that out, promoting it to first, rolling out again, rewriting all Secrets, then removing the old key. Skipping the two-phase rollout means some API server cannot read what another wrote. ## Threat model — say this out loud Encryption at rest defends against: etcd data files, etcd/volume snapshots and backups, and offline disk access. It does **not** defend against anyone who can read the object through the API — the API server decrypts transparently, so a service account with `get secrets` sees plaintext exactly as before. Nor does it help against a compromised API server process, or (for local providers) against a compromised control-plane node where the key file sits. That is why the complete answer is layered: encryption at rest for the storage tier, tight RBAC (especially `list`/`watch` on Secrets) for the API tier, the Node authorizer and NodeRestriction so a kubelet only receives Secrets for its own Pods, and audit policy so Secret reads are recorded. Presenting encryption at rest as "Secrets are now secure" is the failure mode of this answer. ## Managed clusters On EKS/GKE/AKS you typically enable envelope encryption with a cloud KMS key through a cluster setting rather than editing API-server flags; the mechanism underneath is the same KMS provider. Check whether it is on — several providers ship it off by default or only for clusters created after a certain version — and check whether existing Secrets were migrated when it was enabled.