skip to content

Marking Kubernetes ConfigMaps and Secrets immutable is described as a scalability improvement. Describe the mechanism — what work disappears, and where?

level: seniorimportance: nice to knowfreq 22%

answer

  1. kubelet watches every referenced ConfigMap/Secret
  2. default strategy = Watch (also Cache, Get)
  3. immutable → single GET, no watch
  4. cost lands on API server: watchers, cache, re-LIST storm
  5. measure apiserver_registered_watchers

basics

~20 s

By default each kubelet keeps mounted ConfigMaps/Secrets fresh by holding an API watch on every object its Pods reference. Immutable objects can never change, so the kubelet fetches once and opens no watch — removing large numbers of long-lived watches and their memory and CPU cost on the API server.

solid answer

~60 s

The kubelet is responsible for keeping the files it projects into Pods up to date. With the default change-detection strategy (`Watch`), it establishes an individual watch against the API server for every ConfigMap and Secret referenced by a Pod running on that node. Each watch is a long-lived connection plus per-object bookkeeping in the API server's watch cache; multiply by objects per node and nodes per cluster and it becomes one of the larger fixed costs on a big control plane. When an object carries `immutable: true`, the kubelet knows its contents can never change, so it does a single GET, projects the content, and never watches it. The saving lands on the API server — fewer open watches, less cache and less CPU spent fanning out events — and modestly on the kubelet, which stops tracking those objects. The effect is proportional to how many distinct ConfigMaps/Secrets exist and how widely they are spread across nodes. In a cluster with a handful of shared ConfigMaps it is noise; in a multi-tenant cluster with tens of thousands, it is the difference the feature was built for.

code

yaml · 3 lines
yaml
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
configMapAndSecretChangeDetectionStrategy: Watch   # alternatives: Cache, Get

go deeper

for a junior

It is enough to say the kubelet no longer has to watch the object for changes, so the API server holds fewer open watches.

for a middle

Name the default Watch change-detection strategy and explain that watches are per object per node, so the count scales with objects × nodes.

for a senior

Quantify where the cost lands — watch-cache watchers, API-server memory, event fan-out, re-LIST storms after a restart — and say which metrics you would compare before and after.

for a principal

Treat it as a control-plane capacity lever: model the watcher population, decide whether broad adoption is worth the workflow change, and weigh it against alternatives like the Cache detection strategy or reducing per-namespace object sprawl.

## The freshness contract When a Pod mounts a ConfigMap or Secret as a volume, Kubernetes promises eventual freshness: if the object changes, the files inside the container are updated (barring `subPath` mounts, which are not). That promise is what costs money. Somebody has to notice the change and push it down to the node, and that somebody is the kubelet. ## How the kubelet notices, by strategy The kubelet's `ConfigMapAndSecretChangeDetectionStrategy` setting has three modes: - **`Watch`** (the default): for each referenced object, the kubelet opens a watch on the API server and reacts to change events. Freshest, most connections. - **`Cache`**: the kubelet uses a TTL-based cache and re-GETs objects when entries expire — fewer watches, staler data, more periodic GET traffic. - **`Get`**: no cache at all, a GET on every sync loop. Simplest, worst request volume. Nearly every real cluster runs `Watch`. So the steady-state cost is: (objects referenced by Pods on a node) × (nodes) long-lived watches, all terminating at the API server. ## What a watch actually costs the API server A watch is not free bookkeeping. Each one is an open HTTP/2 stream, a registered watcher on the resource's watch cache, and a per-watcher event buffer. The API server maintains a watch cache per resource type; every write to any Secret must be evaluated and dispatched to interested watchers. Memory grows with the number of watchers; CPU grows with writes × watchers-per-object; and each watcher survives API-server restarts only by re-listing, which is why control-plane restarts in big clusters produce a thundering-herd LIST storm. Reducing the watcher count reduces all three, and it especially reduces the re-list stampede. This is why the feature's original design motivation was explicitly scalability, not safety. The KEP framed it as: the kubelet needs to keep contents fresh, freshness is meaningless for objects that never change, so let the object declare that it never changes. ## What immutability removes On seeing `immutable: true`, the kubelet fetches the object once when it is first needed on that node and skips establishing (or tears down) the watch. The projected files are then static for the life of the Pod — which is exactly what the flag already promised. The removed work: - one fewer open watch per (object, node) pair; - the corresponding watch-cache watcher and event buffer on the API server; - the dispatch work whenever anything else in that resource type is written; - the re-LIST on API-server restart or watch expiry. What is *not* removed: the initial GET, the etcd storage of the object itself, and any watches held by non-kubelet clients — controllers, operators, `kubectl get -w`, GitOps agents. Immutability is a hint to the kubelet's object manager, not a cluster-wide suppression of watch traffic. ## When it actually matters Do the arithmetic before claiming a win. A cluster with 50 nodes and 20 shared ConfigMaps has at most a few hundred watches — irrelevant. A multi-tenant platform with 5,000 namespaces, each with its own config and TLS Secrets, spread across 2,000 nodes, can be in the six figures; there the flag is a meaningful lever on API-server memory and on how long a control-plane restart takes to settle. Secondary factors that amplify it: high Pod churn (each new Pod on a new node establishes new watches), heavy use of per-namespace Secrets pulled by many workloads, and clusters where the API server is already the bottleneck rather than etcd. ## How you would measure it The honest senior answer includes the measurement. Look at `apiserver_registered_watchers{group="",kind="ConfigMap"|"Secret"}` and the equivalent for Secrets, plus `apiserver_longrunning_requests`, API-server RSS, and the LIST rate after a control-plane restart. Roll immutability into one large namespace, compare, then decide whether a fleet-wide policy is worth the workflow change. Claiming a percentage improvement without pointing at those series is hand-waving. ## The honest caveat The scalability benefit is real but is rarely the reason a single team adopts the flag — for one application, the change-control benefit dominates. Immutability-for-scale is a **platform-level** decision, applied broadly, and it only pays if adoption is broad enough to shrink the watcher population meaningfully.

  • Does immutability stop controllers and operators from watching the object too?
    No. The flag is honoured by the kubelet's ConfigMap/Secret manager; any other client — a controller, a GitOps agent, `kubectl get -w` — still watches whatever it likes. If your watcher population is dominated by an operator that lists all Secrets cluster-wide, immutability will not move the number.
  • What is the alternative if you cannot make objects immutable but the watch load hurts?
    Switch the kubelet to `ConfigMapAndSecretChangeDetectionStrategy: Cache`, which replaces per-object watches with a TTL cache and periodic GETs. You trade freshness latency and some extra request volume for far fewer long-lived connections. It is a blunter instrument than immutability and it weakens the freshness guarantee for every object on the node.

A mutable ConfigMap is a magazine subscription — the publisher must keep a record of every reader and mail each new issue. An immutable one is a book: you buy it once and nobody has to track you afterwards.

saying these in an interview costs you the question

  • Saying immutability reduces etcd storage — the object is stored identically either way.
  • Claiming it eliminates all watches on the object, including those from controllers and operators.
  • Assuming the benefit is meaningful in a small cluster with a handful of ConfigMaps.
  • Confusing this with the kubelet caching container images or with the API server's own watch cache being disabled.

context

resolving links…