skip to content

In a GitOps pipeline, a team wants to commit their Kubernetes Secret manifests directly into the same git repository that drives deployment (for example via Bitnami's 'Sealed Secrets' controller), including in a repo with broad read access. How can a Secret be safely committed to git, and what does this pattern protect against versus what does it not protect against?

level: principalimportance: nice to knowfreq 30%

answer

  1. kubeseal + public key encrypt, controller private key decrypt
  2. ciphertext safe in git, plaintext Secret unprotected in-cluster
  3. no rotation/audit/expiry - just enables safe git storage
  4. back up the controller's key pair
  5. compare vs External Secrets Operator / Vault

basics

~20 s

Normally you can't put a real password in git because anyone who can read the repo can read it. Sealed Secrets lets you encrypt the password first with a key only your cluster knows, so the encrypted version is safe to commit — only your cluster can turn it back into the real password.

solid answer

~50 s

Sealed Secrets works by having a controller running in-cluster generate an asymmetric key pair; anyone can use the public key, via the `kubeseal` CLI, to encrypt a plaintext Secret into a `SealedSecret` custom resource, which is safe to commit to git because only the controller's private key, which never leaves the cluster, can decrypt it back into a real Kubernetes Secret at apply time. This solves the specific problem of a GitOps repo needing to declaratively include Secret manifests without storing plaintext secrets in git. What it does NOT protect against: once decrypted, the Secret is an ordinary base64-only Secret object inside the cluster — anyone with RBAC read access to Secrets in that namespace can still read it — and it provides no rotation, no per-read audit trail, and no automatic expiry the way Vault's dynamic secrets do.

go deeper

for a junior

Should grasp the basic idea that secrets can be encrypted before being committed to git so the git history itself isn't a leak.

for a middle

Should know the public-key-encrypt / private-key-in-controller-decrypt flow at a conceptual level.

for a senior

Should articulate what this pattern does and doesn't protect, such as in-cluster exposure and no rotation, and compare it to alternatives like External Secrets Operator.

for a principal

Should evaluate this against org-wide compliance and rotation requirements and decide when the git-storable-ciphertext trade-off is acceptable versus when a live-fetch-from-Vault model is required.

## How the pattern works **Sealed Secrets** is a controller-based pattern: a controller deployed in-cluster generates and holds an **asymmetric key pair**, typically rotating it periodically, and publishes only the public half. From there: 1. A developer takes a plaintext Kubernetes Secret manifest and, instead of applying or committing it directly, runs it through the `kubeseal` CLI with that public key, producing a `SealedSecret` custom resource whose data is asymmetrically encrypted ciphertext. 2. This is what gets committed to git and applied through the normal GitOps pipeline, via ArgoCD or Flux syncing the repo. 3. When the SealedSecret manifest is applied to the cluster, the in-cluster controller — the only holder of the private key — decrypts it and materializes a standard Kubernetes Secret object, which the pod then consumes exactly as it would any other Secret. ## The GitOps tension it resolves This exists to resolve a real tension in GitOps. GitOps's core premise is that the git repo is the single declarative source of truth for everything running in the cluster, diffable and auditable via commit history and reviewable via pull request — but plaintext Secrets obviously cannot live safely in git, especially in repos with broad read access such as contractors, CI systems, or forks. Before this pattern, teams either - kept Secrets out of GitOps entirely, applying them manually through a separate, less-audited process and breaking the "everything is declared in git" property, - or used external secret references fetched at apply time from Vault or a cloud secrets manager, which keeps secrets fully out of git but adds a runtime dependency on that external system. Sealed Secrets threads the needle: the encrypted form CAN live in git, satisfying "declare everything", while the plaintext never touches git or any system other than the in-cluster controller. ## What it protects against This protects specifically against the threat of "anyone who can read this git repository's history can read the secret" — since only the private key inside the target cluster can decrypt a SealedSecret, someone who clones the repo, browses old commits, or even has admin access to the git hosting platform gains nothing usable from the ciphertext. - Each SealedSecret is also typically **scoped by default to a specific namespace and name**, so it can't be copy-pasted and reapplied elsewhere in the cluster without re-encryption. - It's also useful for **pull-request review workflows**: a reviewer can see that a secret changed, since a new SealedSecret was committed, and review everything else in the diff without ever seeing the plaintext value. ## What it does not protect against What it explicitly does NOT protect against matters just as much. - **The in-cluster runtime form.** Once the controller decrypts a SealedSecret into a live Kubernetes Secret object inside the cluster, that Secret is a completely ordinary, base64-only-encoded Secret from that point forward — anyone with RBAC permission to read secrets in that namespace, or anyone with access to unencrypted etcd storage, can read the plaintext exactly as with any other Secret. Sealed Secrets adds zero in-cluster protection beyond whatever Kubernetes' normal Secret mechanics already provide. - **Rotation, audit and expiry.** It also provides no rotation, no per-read audit trail, and no automatic expiry — the decrypted value is just as static and long-lived as a normal Secret, unlike a Vault-issued dynamic credential with a TTL and lease-based revocation. - **A compromised controller key.** And it protects nothing if the controller's private key itself is compromised, such as during a cluster compromise, since an attacker with that key can decrypt every SealedSecret ever committed to the repo's history. ## Failure modes in production A common real incident is the controller's private key being lost — for instance a cluster rebuilt without backing up the key — which makes every previously-committed SealedSecret in git permanently undecryptable, forcing every one to be regenerated from the original plaintext and re-sealed with a new key pair; this is why teams are told to back up the controller's key pair as carefully as any other root secret. Another common failure is engineers assuming "it's sealed, so it's fully safe" and relaxing RBAC or logging discipline on the resulting live Secret objects, forgetting the encryption only protects the git-at-rest form, not the in-cluster runtime form. ## The alternatives, and when to prefer them Bitnami's Sealed Secrets project is the most widely used implementation of this pattern, with alternatives including - **Mozilla SOPS**, for encrypting values in git more generally; - **the External Secrets Operator**, which instead syncs Secret values FROM Vault or a cloud secrets manager INTO Kubernetes at apply time, keeping plaintext out of git entirely by never storing it there in any form. A team with strict compliance requirements around credential rotation and per-access auditing, such as PCI-DSS scope, would typically prefer External Secrets Operator plus Vault over Sealed Secrets, precisely because Sealed Secrets solves "safe to commit ciphertext to git" but not "credentials rotate automatically and every read is audited" — a different, and for regulated workloads often more important, problem.

  • If the sealed-secrets controller's private key is lost with no backup, what happens to secrets already running in the cluster versus ones only committed to git so far?
    Secrets already decrypted and materialized as running Kubernetes Secret objects in the cluster are completely unaffected — they keep working as normal Secrets regardless of the sealing key. The problem only hits SealedSecrets not yet applied, or ones that need to be reapplied later during a cluster rebuild or restore — those ciphertexts in git history become permanently undecryptable, and every one must be regenerated from the original plaintext and re-sealed with a new key pair.
  • Why might a team choose External Secrets Operator plus Vault instead of Sealed Secrets, even though both let a GitOps repo declare that a secret exists?
    External Secrets Operator never stores even encrypted secret material in git at all — the repo only references a secret by name or path, and the actual value is fetched live from Vault or a cloud secrets manager at apply time, which supports dynamic, short-TTL credentials, automatic rotation, and per-access audit logging. Sealed Secrets, by contrast, produces a static, long-lived secret once decrypted with none of that rotation or audit machinery, so compliance-heavy environments needing provable rotation and access logs typically prefer the Vault-backed approach.
  • Does scoping a SealedSecret to a specific namespace and name, the default in Bitnami's implementation, matter for security, and why?
    Yes — by default the encryption is bound to the target namespace and Secret name, so even if someone obtained a SealedSecret's ciphertext, they couldn't simply copy-paste and reapply it under a different name or namespace to get a working decryption; the controller enforces that binding at decrypt time. This limits an attacker's ability to relocate or repurpose a stolen ciphertext even within the same cluster, though it does nothing against an attacker who already has genuine access to the original namespace and name.

It's like mailing a locked strongbox through the regular postal service instead of a plaintext letter — mail carriers and anyone who intercepts the box in transit, meaning git history, can't open it, but once it arrives and someone unlocks it at the destination, meaning the cluster, the strongbox's contents sit on a shelf just as exposed as anything else in the room.

saying these in an interview costs you the question

  • Believes SealedSecrets adds encryption protection inside the cluster after decryption
  • Doesn't know the private key lives only in the controller and must be backed up
  • Confuses this pattern with dynamic/rotating secrets like Vault provides
  • Assumes any repo reader with the SealedSecret ciphertext can decrypt it themselves
  • No awareness that losing the controller's key bricks all previously-sealed-but-unapplied secrets

context