A teammate argues that Kubernetes Secret objects are safe to share and commit to git because their values are base64-encoded. What is wrong with that reasoning, and what actually protects Secret data in a cluster?
answer
- base64 = encoding, no key
- default: plaintext in etcd
- tmpfs mount + describe hides values
- list/watch secrets = read them all
- create-pod / exec = indirect read
basics
~20 sBase64 is an encoding, not encryption: anyone decodes it in one command, no key needed. By default a Secret sits unencrypted in etcd and is readable by anyone whose RBAC allows reading Secrets. Real protection is least-privilege RBAC, encryption at rest, and keeping plaintext manifests out of git.
solid answer
~50 sBase64 has no key. `kubectl get secret db -o jsonpath='{.data.password}' | base64 -d` prints the value, so a committed Secret manifest is a committed password. What a Secret actually buys over a ConfigMap is *handling*, not cryptography: `kubectl describe` hides the values, the kubelet mounts them into a memory-backed tmpfs rather than node disk, the Node authorizer limits each kubelet to the Secrets used by Pods on its own node, and `secrets` is a separate RBAC resource you can deny independently of `configmaps`. Three things actually protect the data. First, least-privilege RBAC, remembering that anyone who can create a Pod in a namespace can mount any Secret in that namespace and print it. Second, an `EncryptionConfiguration` on the API server (ideally a KMS provider) so an etcd snapshot is not a password dump. Third, keeping plaintext out of git via SOPS, sealed-secrets, or an external secret manager synced into the cluster.
code
bash · 5 lineskubectl get secret db-creds -o jsonpath='{.data.password}' | base64 -d
# describe hides values, get -o yaml does not
kubectl describe secret db-creds # password: 12 bytes
kubectl get secret db-creds -o yaml # data.password: c3VwZXJzZWNyZXQ=go deeper
State the core fact: base64 is reversible encoding with no key, and Secrets are stored unencrypted in etcd by default. Show the one-line decode.
Add what a Secret genuinely provides over a ConfigMap (tmpfs mount, describe redaction, Node authorizer scoping, separate RBAC resource) and name encryption at rest as the missing piece.
Enumerate the exposure surfaces end to end, including audit-log body capture and the create-Pod/exec escalation paths, and say which control closes each.
Frame it as a key-management question: where does the key live, who can reach it, what is the blast radius of an etcd snapshot, and does the organisation need an external secret manager rather than better hygiene around native Secrets.
## Why Secrets are base64 at all A Kubernetes `Secret` carries its payload in the `data` field as base64 text. Base64 is a *transport encoding*: it maps arbitrary bytes onto a 64-character alphabet so that binary content (a TLS private key, a Kerberos keytab, bytes containing newlines) survives being embedded in JSON and YAML. It takes no key, so it is trivially reversible by anyone who has the bytes. Confidentiality was never its purpose. If you would rather not hand-encode, `stringData` accepts plain UTF-8 on write and the API server converts it into `data` for you. ## What a Secret gives you over a ConfigMap The difference is real but modest, and all of it is about handling rather than cryptography: - `kubectl describe secret` prints `password: 12 bytes` instead of the value, so values do not land in terminal scrollback or CI job logs by accident. - The kubelet writes Secret volumes into a tmpfs (RAM-backed) mount, so the value is not persisted to the node's disk. - The built-in Node authorizer restricts each kubelet to only those Secrets referenced by Pods scheduled on that node, instead of all of them. - The API server can encrypt `secrets` at rest independently of other resource types. - `secrets` is its own RBAC resource, so read access can be denied while `configmaps` stays open. Secret is an access-control boundary with sane defaults. It is not a vault. ## The three exposure surfaces **1. etcd at rest.** In a default installation with no `EncryptionConfiguration`, the serialized object (base64 payload included) is written to etcd verbatim. Anyone with an etcd snapshot, a backup bucket, a stolen disk, or direct etcd client certificates reads every Secret in the cluster without touching the API server or leaving an audit trail. The fix is an `EncryptionConfiguration` passed to `kube-apiserver` via `--encryption-provider-config`, preferably naming a `kms` provider so the key material lives in an external KMS/HSM rather than in a file on the control-plane node. **2. The live API.** Anyone whose RBAC grants `get`, `list`, or `watch` on `secrets` reads the plaintext, because the API server returns the decrypted object. Two indirect paths matter just as much: whoever can create a Pod in a namespace can mount any Secret in that namespace and `cat` it, and whoever can `exec` into an existing Pod inherits whatever credentials that Pod already holds. **3. Git and CI.** A committed Secret manifest is a committed credential, permanently, in every clone and every fork. Rotating the value does not un-leak the old one. Teams solve this with encrypted-at-rest manifests (SOPS, Sealed Secrets) or by not storing the value in the repo at all and letting an operator pull it from a secret manager. A fourth, often-missed surface is the audit log: if the audit policy records `RequestBody` or `Response` for `secrets`, the values are copied into every audit sink. The conventional policy pins `secrets`, `configmaps`, and `tokenreviews` to `level: Metadata`. ## After delivery Once a Secret reaches a container the protection is whatever the app does with it. Values injected as environment variables are visible in `/proc/<pid>/environ`, are inherited by every child process, and are routinely captured wholesale by crash reporters and error-tracking SDKs. Values printed into logs are the single most common real-world leak, and log pipelines usually have far wider read access than the `secrets` resource does. ## How to answer Say plainly that base64 is encoding, not encryption; name the default (unencrypted in etcd); then list the three controls that matter: RBAC scoped tightly (including the create-Pod and exec escalation paths), encryption at rest with a KMS provider, and plaintext never entering version control.
- If base64 offers nothing, why does the Secret API use it instead of a plain string field?Because Secret values are arbitrary byte arrays, not necessarily valid UTF-8 text: TLS private keys, keytabs, and binary tokens all need to round-trip through JSON and YAML unchanged. Base64 guarantees that. The `stringData` write-only field exists for the common UTF-8 case and is converted into `data` on write.
- A developer has no RBAC access to Secrets but can create Deployments in the namespace. Can they read a Secret there?Yes. They can create a Pod that mounts the Secret as a volume or as environment variables and then read it from inside the container, or simply echo it to the Pod's logs which they can read. Namespace-scoped Pod-create permission is effectively read access to every Secret in that namespace, which is why namespace boundaries, not RBAC verbs alone, are the real isolation unit for Secrets.
Base64 is a shipping crate, not a safe: it keeps odd-shaped cargo from breaking in transit, and anyone can pry the lid with their hands.
saying these in an interview costs you the question
- Calling base64 'encryption' or saying Secrets are 'encrypted by default'
- Believing a Secret is safe in git because it is not human-readable at a glance
- Thinking `kubectl describe` hiding the value means the value is protected on the wire or in etcd
- Assuming denying the `secrets` RBAC resource is sufficient, ignoring the Pod-create and exec escalation paths