Kubernetes workloads need Secrets, but plaintext secrets cannot be committed. Compare committing encrypted secrets to Git (SOPS, Sealed Secrets) with having an in-cluster operator fetch values from an external secret manager.

seniorshould knowfreq 50%

answer

Git history is forever
who holds the decryption key
cluster-bound ciphertext, cluster-bound recovery
a reference is not a value
rotation without a commit

basics

~20 s

Encrypting secrets into Git keeps one source of truth and works for bootstrap, but rotation needs a commit and old ciphertext lives in history forever. An operator fetching from an external manager keeps only a reference in Git, so rotation is external — at the cost of a live runtime dependency.

solid answer

~50 s

Both approaches exist because a GitOps agent can only apply what Git declares, and Git is a widely-cloned, permanently-retained store. SOPS encrypts the values inside a YAML file with a KMS or age key, so the file stays reviewable and the ciphertext is decrypted at sync time. Sealed Secrets encrypts with a specific cluster's controller public key using kubeseal, so the ciphertext only decrypts in that cluster — excellent isolation, but the sealing key becomes disaster-recovery state you must back up, and a sealed value cannot be reused in another cluster. External Secrets Operator inverts it: Git holds an ExternalSecret naming a store and a remote key, so rotation happens in the manager with no commit at all, but Git no longer describes the full desired state and the cluster needs live credentials and connectivity. Teams often bootstrap with an encrypted-in-Git option and move application secrets to the operator once a manager exists.

Both approaches exist because a GitOps agent can only apply what Git declares, and Git is a widely-cloned, permanently-retained store. **SOPS** encrypts the values inside a YAML file with a KMS or age key, so the file stays reviewable and the ciphertext is decrypted at sync time. **Sealed Secrets** encrypts with a specific cluster's controller public key using `kubeseal`, so the ciphertext only decrypts in that cluster — excellent isolation, but the sealing key becomes disaster-recovery state you must back up, and a sealed value cannot be reused in another cluster. **External Secrets Operator** inverts it: Git holds an `ExternalSecret` naming a store and a remote key, so rotation happens in the manager with no commit at all, but Git no longer describes the full desired state and the cluster needs live credentials and connectivity. Teams often bootstrap with an encrypted-in-Git option and move application secrets to the operator once a manager exists.

code

yaml · 17 lines

yaml

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: checkout-db
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: production-store
    kind: ClusterSecretStore
  target:
    name: checkout-db
  data:
    - secretKey: password
      remoteRef:
        key: production/checkout/db
        property: password

go deeper

for a junior

Know that a Kubernetes Secret is only base64-encoded, so it must never be committed as-is, and that the standard options are committing an encrypted form or committing a reference the cluster resolves.

for a middle

Explain the mechanics: SOPS encrypts values under a KMS or age key and is decrypted at sync; kubeseal encrypts to a specific cluster's controller key; the External Secrets operator materializes a Secret from a remote store on a refresh interval.

for a senior

Reason about the failure modes you have lived through — the sealing key missing during cluster rebuild, a manager outage blocking Pod startup, and rotation that re-encrypts the same compromised value instead of changing it.

for a principal

Own the policy: who is allowed to decrypt and where that is audited, how bootstrap credentials are minimized, whether secrets may live in a source repository at all, and how the choice scales across clusters and tenants without one key unlocking the estate.

The constraint that creates the problem

A GitOps agent applies whatever the repository declares. A Kubernetes Secret is base64-encoded, not encrypted, so committing one is committing plaintext. Git makes this worse than an ordinary bad file: history is retained, cloned widely, mirrored to CI runners, and effectively impossible to purge. Any answer has to say where the ciphertext or the reference lives and who holds the decryption key.

Option one: encrypt the values in place (SOPS)

SOPS encrypts the values of a YAML or JSON document while leaving the keys readable, using a data key wrapped by a KMS key, an age key or PGP. A .sops.yaml in the repository maps path patterns to the keys used. The committed file is reviewable — you can see that DB_PASSWORD changed even though you cannot read it — and diffs remain meaningful at the structural level.

Decryption happens at sync time. Flux supports SOPS decryption natively in its Kustomization, referencing a key stored as a Kubernetes Secret; Argo CD needs a config-management plugin to do the same. Access control is the KMS key policy, which is a real advantage: you can grant and revoke decryption without touching the repository, and cloud KMS gives you an audit log of decryptions.

Weaknesses. Rotating a value is a commit, so secret rotation inherits pull-request latency. Old ciphertext stays in history forever, so a leaked key retroactively exposes every historical value — rotation must therefore mean changing the secret, not just re-encrypting it. And whoever can decrypt in CI can decrypt everything that key covers, so key scoping needs the same care as the repository layout.

Option two: encrypt to a specific cluster (Sealed Secrets)

The kubeseal CLI encrypts a Secret with the public key of the Sealed Secrets controller running in the target cluster, producing a SealedSecret custom resource that is safe to commit. Only that controller's private key can decrypt it, and by default the encryption is scoped to the resource's name and namespace, so the ciphertext cannot be replayed into a different namespace.

That scoping is the strength and the weakness. Cluster-scoped ciphertext means a stolen repository is useless without that cluster — but it also means the sealing key is now critical recovery state. Rebuilding a cluster without restoring the sealing key leaves every committed SealedSecret undecryptable, and every value must be re-sealed. Multi-cluster estates must either re-seal per cluster or share one key pair across clusters, which gives back the isolation that made the approach attractive.

Option three: keep only a reference (External Secrets Operator)

Here the repository holds an ExternalSecret naming a SecretStore or ClusterSecretStore and the remote key path; the operator reads the value from the manager and materializes a normal Kubernetes Secret, refreshing on an interval.

What this buys. No ciphertext in Git at all, so no historical exposure. Rotation happens in the manager and propagates on the next refresh with no commit and no deployment. The manager's own audit log and access policies apply. It also fits organizations where a security team already owns secret storage and will not accept secrets living in a source repository under any encryption.

What it costs. Git no longer describes the full desired state — the cluster's actual configuration now depends on something outside the repository, which weakens the reproducibility argument for GitOps. There is a live runtime dependency: if the manager is unreachable, new Pods needing a not-yet-materialized Secret cannot start. And you have a bootstrap chicken-and-egg — the credential the operator uses to authenticate cannot itself be an ExternalSecret, so it comes from workload identity where available, or from one of the encrypted-in-Git approaches.

Choosing

A reasonable default for a team with no secret manager: SOPS with a cloud KMS key, because it needs no extra runtime component and the key policy is real access control. Sealed Secrets suits a single cluster where per-cluster binding is a feature and the sealing key can be backed up properly. Once a managed secret store exists and is operated by someone, moving application secrets to the operator is usually right, keeping exactly one encrypted-in-Git secret: the operator's own bootstrap credential.

Things that go wrong regardless

Rotating by re-encrypting the same value. The value is what leaked; re-wrapping it changes nothing.
Committing a plaintext secret once and "fixing" it in a later commit — history retains it, so the secret is burned and must be rotated.
Ciphertext in Git still leaks the shape: key names, which environments hold which credentials, when a credential last changed.
Materialized Secrets are still ordinary Kubernetes Secrets; anyone with read permission on the namespace can read them however they arrived.

## The constraint that creates the problem A GitOps agent applies whatever the repository declares. A Kubernetes `Secret` is base64-encoded, not encrypted, so committing one is committing plaintext. Git makes this worse than an ordinary bad file: history is retained, cloned widely, mirrored to CI runners, and effectively impossible to purge. Any answer has to say where the *ciphertext or the reference* lives and who holds the decryption key. ## Option one: encrypt the values in place (SOPS) SOPS encrypts the values of a YAML or JSON document while leaving the keys readable, using a data key wrapped by a KMS key, an age key or PGP. A `.sops.yaml` in the repository maps path patterns to the keys used. The committed file is reviewable — you can see that `DB_PASSWORD` changed even though you cannot read it — and diffs remain meaningful at the structural level. Decryption happens at sync time. Flux supports SOPS decryption natively in its `Kustomization`, referencing a key stored as a Kubernetes Secret; Argo CD needs a config-management plugin to do the same. Access control is the KMS key policy, which is a real advantage: you can grant and revoke decryption without touching the repository, and cloud KMS gives you an audit log of decryptions. **Weaknesses.** Rotating a value is a commit, so secret rotation inherits pull-request latency. Old ciphertext stays in history forever, so a leaked key retroactively exposes every historical value — rotation must therefore mean *changing the secret*, not just re-encrypting it. And whoever can decrypt in CI can decrypt everything that key covers, so key scoping needs the same care as the repository layout. ## Option two: encrypt to a specific cluster (Sealed Secrets) The `kubeseal` CLI encrypts a Secret with the public key of the Sealed Secrets controller running in the target cluster, producing a `SealedSecret` custom resource that is safe to commit. Only that controller's private key can decrypt it, and by default the encryption is scoped to the resource's name and namespace, so the ciphertext cannot be replayed into a different namespace. That scoping is the strength and the weakness. Cluster-scoped ciphertext means a stolen repository is useless without that cluster — but it also means the sealing key is now critical recovery state. Rebuilding a cluster without restoring the sealing key leaves every committed `SealedSecret` undecryptable, and every value must be re-sealed. Multi-cluster estates must either re-seal per cluster or share one key pair across clusters, which gives back the isolation that made the approach attractive. ## Option three: keep only a reference (External Secrets Operator) Here the repository holds an `ExternalSecret` naming a `SecretStore` or `ClusterSecretStore` and the remote key path; the operator reads the value from the manager and materializes a normal Kubernetes `Secret`, refreshing on an interval. **What this buys.** No ciphertext in Git at all, so no historical exposure. Rotation happens in the manager and propagates on the next refresh with no commit and no deployment. The manager's own audit log and access policies apply. It also fits organizations where a security team already owns secret storage and will not accept secrets living in a source repository under any encryption. **What it costs.** Git no longer describes the full desired state — the cluster's actual configuration now depends on something outside the repository, which weakens the reproducibility argument for GitOps. There is a live runtime dependency: if the manager is unreachable, new Pods needing a not-yet-materialized Secret cannot start. And you have a bootstrap chicken-and-egg — the credential the operator uses to authenticate cannot itself be an `ExternalSecret`, so it comes from workload identity where available, or from one of the encrypted-in-Git approaches. ## Choosing A reasonable default for a team with no secret manager: SOPS with a cloud KMS key, because it needs no extra runtime component and the key policy is real access control. Sealed Secrets suits a single cluster where per-cluster binding is a feature and the sealing key can be backed up properly. Once a managed secret store exists and is operated by someone, moving application secrets to the operator is usually right, keeping exactly one encrypted-in-Git secret: the operator's own bootstrap credential. ## Things that go wrong regardless - Rotating by re-encrypting the same value. The value is what leaked; re-wrapping it changes nothing. - Committing a plaintext secret once and "fixing" it in a later commit — history retains it, so the secret is burned and must be rotated. - Ciphertext in Git still leaks the *shape*: key names, which environments hold which credentials, when a credential last changed. - Materialized Secrets are still ordinary Kubernetes Secrets; anyone with read permission on the namespace can read them however they arrived.

A plaintext secret was committed and removed in the next commit. Is that fixed?
No. The value remains in history, in every clone, fork, mirror and CI cache, and rewriting history does not reach copies already pulled. The only real fix is to treat the credential as compromised and rotate it at the source, then decide separately whether history rewriting is worth the disruption.
With External Secrets, how does the operator itself authenticate without a chicken-and-egg problem?
Preferably with workload identity — the cluster's service account is federated to a cloud role, so no static credential exists at all. Where that is unavailable, exactly one bootstrap credential is committed under SOPS or sealed to the cluster, and everything else references the manager. Keeping that to a single, narrowly-scoped credential is the point.
What does encrypted-in-Git still reveal to someone who clones the repository?
The structure: which secrets exist, their key names, which environments and namespaces hold which credentials, and when each value last changed. That metadata is genuine reconnaissance value even without plaintext, which is a reason to keep secret manifests out of repositories that are more widely readable than the systems they unlock.

saying these in an interview costs you the question

Says base64 in a Kubernetes Secret is encryption
Rotates a leaked secret by re-encrypting the same value
Forgets the Sealed Secrets sealing key in disaster recovery
Assumes External Secrets keeps Git the full source of truth
Thinks deleting the commit removes the secret from history

Kubernetes workloads need Secrets, but plaintext secrets cannot be committed. Compare committing encrypted secrets to Git (SOPS, Sealed Secrets) with having an in-cluster operator fetch values from an external secret manager.

§ answer

basics

solid answer

code

§ go deeper

▾level notes4 levels

▾deep dive~4 min

The constraint that creates the problem

Option one: encrypt the values in place (SOPS)

Option two: encrypt to a specific cluster (Sealed Secrets)

Option three: keep only a reference (External Secrets Operator)

Choosing

Things that go wrong regardless

▾common follow-ups3 questions

▾red flags5 flags

§ context

▾related questions4 questions

▾related topics4 topics

The constraint that creates the problem

Option one: encrypt the values in place (SOPS)

Option two: encrypt to a specific cluster (Sealed Secrets)

Option three: keep only a reference (External Secrets Operator)

Choosing

Things that go wrong regardless

answer

go deeper

context