skip to content

Your team deploys to Kubernetes with GitOps: how can Secret values live in Git safely, and how do Sealed Secrets and SOPS differ?

level: juniorimportance: should knowfreq 52%

answer

  1. base64 in Git is plaintext
  2. encrypt before commit
  3. cluster-held private key vs external master key
  4. values encrypted, keys readable
  5. scope binds name and namespace

basics

~20 s

Never commit a plain Secret manifest, because base64 is readable. Sealed Secrets commits a SealedSecret that only an in-cluster controller can decrypt. SOPS commits values encrypted with an external key and decrypts them at deploy time.

solid answer

~50 s

GitOps wants every object in Git, but a Kubernetes `Secret` manifest only base64-encodes its values, so committing one leaks the credential to everyone who can read the repository. There are two common fixes. **Sealed Secrets** runs a controller in the cluster that holds a private key. You encrypt locally with `kubeseal` against its public certificate and commit a `SealedSecret` (`bitnami.com/v1alpha1`), and the controller decrypts it into a normal `Secret`. **SOPS** encrypts only the *values* of a YAML file with a key held in a cloud KMS, age or PGP. The file stays diffable, and whatever applies the manifests must be able to decrypt it. Both still put ciphertext in Git, so the key becomes the crown jewel. A third option commits no secret material at all: commit an `ExternalSecret` that points at a secret manager.

code

bash · 7 lines
bash
kubectl create secret generic tracking-db \
  --namespace shipment-tracking \
  --from-literal=password='s3cr3t-8841' \
  --dry-run=client -o yaml \
  | kubeseal --format yaml > tracking-db-sealed.yaml

git add tracking-db-sealed.yaml

go deeper

for a junior

Remember that base64 is not encryption, so a Secret manifest must not be committed. Name Sealed Secrets and SOPS as the two ways to commit an encrypted form instead.

for a middle

Explain who decrypts in each model: the in-cluster controller with its private key, or the delivery tooling with the master key. Also explain how Sealed Secrets scopes bind the name and namespace.

for a senior

Show that you have operated this: back up the sealing keys for disaster recovery, understand that key renewal is not value rotation, and know how a leaked value is actually revoked.

for a principal

Frame the choice as where trust lives: a cluster-held key, an external master key, or no secret material in Git at all behind a secret manager. Weigh that against how many clusters and teams you run.

## The problem GitOps creates **GitOps** means the desired state of the cluster lives in a Git repository, and a controller applies whatever is committed. That works well for Deployments and Services, but it clashes with credentials. A Kubernetes `Secret` stores its values under `data` as **base64**, which is an encoding, not encryption. Anyone who can clone the repository can decode a committed Secret in one command, and Git history keeps the value forever, even after the file is deleted. So a team that wants "everything in Git" needs one of three approaches: 1. Commit an **encrypted** form of the Secret that only the cluster can open (Sealed Secrets). 2. Commit a file whose **values are encrypted** with an external key, and decrypt it during deployment (SOPS). 3. Commit only a **reference** to a value that lives in an external secret manager, and let an operator fetch it (the External Secrets Operator). ## Sealed Secrets **Sealed Secrets** adds a custom resource, `SealedSecret`, and a controller that runs in the cluster. - At startup the controller generates an asymmetric key pair and stores the private key in a Secret in its own namespace, labelled `sealedsecrets.bitnami.com/sealed-secrets-key`. - A developer runs `kubeseal`, which fetches the controller's public certificate (or reads one given with `--cert`) and encrypts a local Secret into a `SealedSecret` with the values under `spec.encryptedData`. - The `SealedSecret` is safe to commit. Only the controller holds the private key, so only the controller can decrypt it into a real `Secret` in the target namespace. - **Scope** is bound into the ciphertext. The default `strict` scope ties it to one name and one namespace, so a copied and renamed SealedSecret fails to decrypt. `namespace-wide` allows renaming within the namespace, and `cluster-wide` allows any name and namespace. You pick the scope with `kubeseal --scope` or with the `sealedsecrets.bitnami.com/namespace-wide` and `sealedsecrets.bitnami.com/cluster-wide` annotations. - By default the controller adds a **new sealing key every 30 days** (`--key-renew-period`). It keeps the old keys, so existing SealedSecrets still decrypt. Renewal does not change any secret *value*, and existing objects are not re-sealed unless you run `kubeseal --re-encrypt`. The operational catch is that the private keys are now the most sensitive thing you own. If they are lost in a cluster rebuild, every SealedSecret in Git becomes unreadable, so they must be backed up out of band. ## SOPS **SOPS** is a file-encryption tool rather than a Kubernetes controller. - It encrypts the **values** of a YAML or JSON file and leaves the keys and structure readable, so pull-request diffs still show *which* field changed. - The data key is protected by one or more master keys: a cloud KMS key, an age key or a PGP key. - Something in the delivery path must decrypt before or during apply. That is either a decryption step built into the GitOps controller, a plugin, or a CI job. The cluster never sees the SOPS file itself. - Access control moves to whoever can use the master key, which is usually easier to audit and revoke than a key stored in the cluster. ## Comparing the options | Aspect | Sealed Secrets | SOPS | External Secrets Operator | |---|---|---|---| | What is in Git | `SealedSecret` ciphertext | Encrypted values in a normal manifest | An `ExternalSecret` reference, with no secret material | | Who decrypts | The in-cluster controller | The delivery tooling that holds the master key | Nobody. The value is read from the provider | | Key to protect | The controller's private sealing keys | The KMS, age or PGP master key | The operator's credential to the provider | | Rotating a value | Re-seal and commit | Re-encrypt and commit | Change it in the provider and wait for the refresh | | Works without a secret manager | Yes | Yes (age or PGP) | No | ## How to choose - Small team with **no secret manager**: Sealed Secrets is the least moving parts, provided the sealing keys are backed up. - Several clusters or environments where people **review encrypted diffs**: SOPS fits well, and a KMS-backed key gives central revocation. - An organisation that already runs **a secret manager**: commit `ExternalSecret` objects instead. Git then holds no ciphertext at all, and rotation needs no commit. Whichever you choose, remember that the result in the cluster is still an ordinary `Secret`. Protecting it at rest and restricting who can read it is a separate job.

  • The cluster is rebuilt from scratch and every SealedSecret fails to decrypt. What went wrong?
    The new controller generated new key pairs, and the old private keys that the committed SealedSecrets were encrypted against were not restored. The fix is to back up the controller's key Secrets and restore them before the controller starts, or to re-seal every secret against the new certificate. Treat those keys as disaster-recovery material.
  • Can't a manifest generator just build the Secret from a local file at deploy time instead?
    It can, but the file still has to exist somewhere. Either it is committed in plaintext, or it sits on a build machine outside version control and audit. Generators package values you already hold. They do not fetch them. To pull real secret material from a secret manager at runtime, use an operator such as the External Secrets Operator or the Secrets Store CSI Driver.
  • Does Sealed Secrets' 30-day key renewal rotate your passwords?
    No. Renewal only adds a new sealing key for future `kubeseal` runs, and old keys are kept so existing SealedSecrets keep decrypting. The credential inside is unchanged. Rotating a password means generating a new value, re-sealing it and committing it, and existing objects are re-sealed only if you run `kubeseal --re-encrypt`.

Sealed Secrets is a drop box with a slot: anyone can post a sealed envelope, but only the bank inside holds the key that opens it.

saying these in an interview costs you the question

  • Committing a Secret is fine because base64 hides the value
  • Anyone with kubeseal's public certificate can decrypt a SealedSecret
  • Sealing key renewal re-encrypts every SealedSecret automatically
  • SOPS encrypts the whole file, so diffs become unreadable
  • A private Git repository makes plaintext Secrets acceptable
  • Losing the controller's private key only affects new secrets