Would you require every ConfigMap and Secret in a large shared Kubernetes cluster to be created with `immutable: true`? Argue both sides and say where you would land.

principalnice to knowfreq 14%

answer

config changes become versioned releases
kubelet watch relief only pays at fleet scale
carve out controller-written Secrets (cert-manager, ESO)
retention N versions or rollback dies
enforce in the generator, policy as backstop

basics

~20 s

Usually yes for application config, enforced by generation tooling rather than by policy alone — but with carve-outs for objects written by controllers (TLS renewal, external-secret sync), a retention policy so rollback still works, and an agreed break-glass procedure for incidents.

solid answer

~60 s

I would default to immutable and treat exceptions as deliberate.

For: config changes become versioned, revertible rollouts instead of invisible in-place edits; a whole class of incidents ("someone edited the shared ConfigMap") disappears; and at fleet scale the kubelet stops holding a watch per object per node, which is real API-server relief.

Against: it costs a workflow. Every change produces a new object, so you need hash-based naming, reference rewriting, and garbage collection or etcd fills with orphans. Break-glass edits during an incident become delete-and-recreate, which is disruptive. And it is incompatible with anything that writes Secrets in place — cert-manager renewing a kubernetes.io/tls Secret, External Secrets Operator syncing from Vault, service-account token controllers.

So: mandate it where a pipeline generates the object (Kustomize configMapGenerator, Helm with templated names) and enforce it by making the generator emit the flag, not by rejecting non-immutable objects with an admission policy. Exempt controller-owned Secrets by label. Retain the last N versions per workload so rollout undo remains real, and document the delete-recreate break-glass path.

I would default to immutable and treat exceptions as deliberate. **For**: config changes become versioned, revertible rollouts instead of invisible in-place edits; a whole class of incidents ("someone edited the shared ConfigMap") disappears; and at fleet scale the kubelet stops holding a watch per object per node, which is real API-server relief. **Against**: it costs a workflow. Every change produces a new object, so you need hash-based naming, reference rewriting, and garbage collection or etcd fills with orphans. Break-glass edits during an incident become delete-and-recreate, which is disruptive. And it is incompatible with anything that writes Secrets in place — cert-manager renewing a `kubernetes.io/tls` Secret, External Secrets Operator syncing from Vault, service-account token controllers. So: mandate it where a pipeline generates the object (Kustomize `configMapGenerator`, Helm with templated names) and enforce it by making the generator emit the flag, not by rejecting non-immutable objects with an admission policy. Exempt controller-owned Secrets by label. Retain the last N versions per workload so `rollout undo` remains real, and document the delete-recreate break-glass path.

code

yaml · 22 lines

yaml

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-immutable-config
spec:
  validationFailureAction: Audit   # start in Audit, promote to Enforce after inventory
  rules:
    - name: configmaps-and-secrets-must-be-immutable
      match:
        any:
          - resources:
              kinds: [ConfigMap, Secret]
      exclude:
        any:
          - resources:
              selector:
                matchLabels:
                  config.platform/mutable-owner: "*"
      validate:
        message: "ConfigMaps and Secrets must set immutable: true (or carry a mutable-owner label)"
        pattern:
          immutable: true

go deeper

for a junior

Focus on the concrete effects you understand: edits are blocked, changes ship as new objects, and old ones must be cleaned up.

for a middle

Give the two-sided trade — safer versioned rollouts versus more objects and a harder emergency edit — and note that the tooling has to generate the names.

for a senior

Lead with operational reality: which controllers break, how rollback survives pruning, what the break-glass path is, and how you would stage the rollout.

for a principal

Own the whole policy: enforcement point (generator vs admission backstop), exemption model, retention and GC, measured justification, and the sequencing that avoids breaking controller-written Secrets on day one.

Framing the decision

This is a platform policy question, not a Kubernetes trivia question. The flag itself is trivial; what you are really deciding is whether the organisation's configuration changes go through a release process. Answer it as a trade of operational flexibility for change control plus control-plane capacity.

The case for a default-immutable policy

Change control. A mutable ConfigMap mounted by fifty Pods is a global variable with no version history. Editing it changes behaviour across the fleet with no rollout, no rollout status, no rollout undo, and an audit trail that lives only in the API-server audit log. Immutability converts every config change into a new object name, hence a Pod-template change, hence a normal rolling update with all the release machinery attached — including a rollback that actually works because the previous object still exists.

Incident class removed. "Config changed under running Pods and half of them picked it up because they restarted" is a genuinely nasty failure mode: the same Deployment, same image, two behaviours depending on Pod age. Immutability makes it structurally impossible.

Control-plane capacity. The kubelet holds a watch per referenced ConfigMap/Secret per node under its default detection strategy; immutable objects are fetched once and not watched. In a cluster with thousands of namespaces this is a meaningful reduction in registered watchers, API-server memory, and the LIST storm after a control-plane restart. Note this benefit only materialises with broad adoption — which is precisely why it is argued as a fleet policy rather than per team.

The case against

Workflow cost is real and lands on every team. Someone must generate names, rewrite references and prune. If you mandate the flag without shipping the tooling, teams will hand-roll -v2, -v3, -v3-final suffixes, collide with each other, and the policy will be resented and worked around.

Break-glass gets slower. At 03:00 with a bad feature flag, kubectl edit + restart is a 30-second fix. Immutable makes it create-new + apply + roll, or the disruptive kubectl replace --force (which deletes the object, so any Pod starting in that gap lands in CreateContainerConfigError). That is usually the right trade — the fast path is also the untracked path — but it must be a decision, documented, not a surprise during an incident.

Controller compatibility. A large share of Secrets in a real cluster are written by controllers that update in place: cert-manager renewing TLS material, External Secrets Operator syncing from a secret manager, registry-credential refreshers, service-account token controllers. Immutability breaks their write path. Any blanket mandate must carve these out, and the carve-out has to be by ownership label or namespace, not by hope.

Orphan accumulation. Without pruning, the etcd savings invert: you traded watches for a growing pile of dead objects that every LIST must page through. The policy is incomplete without a retention rule.

Where I land, and how I would enforce it

Default immutable for pipeline-generated application config, enforced at the point of generation:

Kustomize configMapGenerator/secretGenerator with hash suffixes and options: {immutable: true}; Helm charts with templated, version-bearing names. The flag is emitted by the tool, so nobody has to remember it.
Admission policy (ValidatingAdmissionPolicy or Gatekeeper/Kyverno) used as a backstop with an exemption label such as config.platform/mutable-owner: cert-manager, not as the primary mechanism. Policy that blocks controllers is how you cause an outage on rollout day.
Retention: keep the last N (3–5) versions per workload; prune only objects no live ReplicaSet references. Deleting the previous version instantly is the single most common way teams accidentally destroy their own rollback.
Ownership/GC: give generated objects owner references where a natural owner exists, so cleanup is automatic rather than a cron job.
A written break-glass procedure: who may delete-and-recreate, what the blast radius is, and the requirement to reconcile the repo afterwards so GitOps does not revert the fix.

Sequencing the rollout

Do not flip the fleet at once. Inventory who writes Secrets in place (kubectl get secrets -A grouped by owning controller / managed fields). Pilot one large namespace, measure apiserver_registered_watchers and post-restart LIST behaviour before and after, and confirm the deploy tooling's prune semantics do not delete a version an active ReplicaSet still needs. Then widen. If the measurement shows no control-plane benefit at your scale, the policy still stands on the change-control argument alone — but be honest about which argument is doing the work.

The answer that fails

"Yes, immutable everywhere, it's a best practice" — with no mention of controller-written Secrets, no pruning story, and no break-glass path. That is a policy that gets rolled back after its first incident.

## Framing the decision This is a platform policy question, not a Kubernetes trivia question. The flag itself is trivial; what you are really deciding is whether the organisation's configuration changes go through a release process. Answer it as a trade of operational flexibility for change control plus control-plane capacity. ## The case for a default-immutable policy **Change control.** A mutable ConfigMap mounted by fifty Pods is a global variable with no version history. Editing it changes behaviour across the fleet with no rollout, no `rollout status`, no `rollout undo`, and an audit trail that lives only in the API-server audit log. Immutability converts every config change into a new object name, hence a Pod-template change, hence a normal rolling update with all the release machinery attached — including a rollback that actually works because the previous object still exists. **Incident class removed.** "Config changed under running Pods and half of them picked it up because they restarted" is a genuinely nasty failure mode: the same Deployment, same image, two behaviours depending on Pod age. Immutability makes it structurally impossible. **Control-plane capacity.** The kubelet holds a watch per referenced ConfigMap/Secret per node under its default detection strategy; immutable objects are fetched once and not watched. In a cluster with thousands of namespaces this is a meaningful reduction in registered watchers, API-server memory, and the LIST storm after a control-plane restart. Note this benefit only materialises with **broad** adoption — which is precisely why it is argued as a fleet policy rather than per team. ## The case against **Workflow cost is real and lands on every team.** Someone must generate names, rewrite references and prune. If you mandate the flag without shipping the tooling, teams will hand-roll `-v2`, `-v3`, `-v3-final` suffixes, collide with each other, and the policy will be resented and worked around. **Break-glass gets slower.** At 03:00 with a bad feature flag, `kubectl edit` + restart is a 30-second fix. Immutable makes it create-new + apply + roll, or the disruptive `kubectl replace --force` (which deletes the object, so any Pod starting in that gap lands in `CreateContainerConfigError`). That is usually the right trade — the fast path is also the untracked path — but it must be a decision, documented, not a surprise during an incident. **Controller compatibility.** A large share of Secrets in a real cluster are written by controllers that update in place: cert-manager renewing TLS material, External Secrets Operator syncing from a secret manager, registry-credential refreshers, service-account token controllers. Immutability breaks their write path. Any blanket mandate must carve these out, and the carve-out has to be by ownership label or namespace, not by hope. **Orphan accumulation.** Without pruning, the etcd savings invert: you traded watches for a growing pile of dead objects that every LIST must page through. The policy is incomplete without a retention rule. ## Where I land, and how I would enforce it Default immutable for **pipeline-generated application config**, enforced at the point of generation: - Kustomize `configMapGenerator`/`secretGenerator` with hash suffixes and `options: {immutable: true}`; Helm charts with templated, version-bearing names. The flag is emitted by the tool, so nobody has to remember it. - Admission policy (ValidatingAdmissionPolicy or Gatekeeper/Kyverno) used as a **backstop with an exemption label** such as `config.platform/mutable-owner: cert-manager`, not as the primary mechanism. Policy that blocks controllers is how you cause an outage on rollout day. - Retention: keep the last N (3–5) versions per workload; prune only objects no live ReplicaSet references. Deleting the previous version instantly is the single most common way teams accidentally destroy their own rollback. - Ownership/GC: give generated objects owner references where a natural owner exists, so cleanup is automatic rather than a cron job. - A written break-glass procedure: who may delete-and-recreate, what the blast radius is, and the requirement to reconcile the repo afterwards so GitOps does not revert the fix. ## Sequencing the rollout Do not flip the fleet at once. Inventory who writes Secrets in place (`kubectl get secrets -A` grouped by owning controller / managed fields). Pilot one large namespace, measure `apiserver_registered_watchers` and post-restart LIST behaviour before and after, and confirm the deploy tooling's prune semantics do not delete a version an active ReplicaSet still needs. Then widen. If the measurement shows no control-plane benefit at your scale, the policy still stands on the change-control argument alone — but be honest about which argument is doing the work. ## The answer that fails "Yes, immutable everywhere, it's a best practice" — with no mention of controller-written Secrets, no pruning story, and no break-glass path. That is a policy that gets rolled back after its first incident.

Which workloads would you exempt outright?
Anything whose Secret is written in place by a controller: cert-manager-managed `kubernetes.io/tls` Secrets that get renewed, External Secrets Operator or Secrets Store CSI sync targets, registry-credential refreshers, and service-account token Secrets. Exempt them by an ownership label so the exemption is explicit and auditable rather than a hole in the policy.
How would you prove the control-plane benefit rather than assert it?
Pilot on one large namespace and compare `apiserver_registered_watchers` for ConfigMaps and Secrets, `apiserver_longrunning_requests`, API-server RSS, and the LIST rate and settle time after a control-plane restart. If the numbers barely move at your scale, keep the policy for its change-control value and stop citing scalability as the justification.

saying these in an interview costs you the question

Mandating immutability cluster-wide without inventorying the controllers that write Secrets in place.
Presenting the API-server watch saving as a benefit at small scale where it is negligible.
Having no retention policy, so pruning either fills etcd with orphans or destroys the ability to roll back.
Enforcing with a hard admission deny on day one instead of Audit-then-Enforce with exemptions.
Treating `kubectl replace --force` as an acceptable routine workflow rather than a break-glass action.

Would you require every ConfigMap and Secret in a large shared Kubernetes cluster to be created with `immutable: true`? Argue both sides and say where you would land.

§ answer

basics

solid answer

code

§ go deeper

▾level notes4 levels

▾deep dive~4 min

Framing the decision

The case for a default-immutable policy

The case against

Where I land, and how I would enforce it

Sequencing the rollout

The answer that fails

▾common follow-ups2 questions

▾red flags5 flags

§ context

▾related questions4 questions

▾related topics3 topics

Framing the decision

The case for a default-immutable policy

The case against

Where I land, and how I would enforce it

Sequencing the rollout

The answer that fails

answer

go deeper

context