As a cluster grows to thousands of nodes and dozens of controllers, watch traffic becomes a significant load on the control plane. How do you reason about that cost and reduce it on the client side?

## Where the cost actually is A watch is cheap while idle and expensive under churn. Three distinct costs matter. **Fan-out on change.** When an object changes, the API server must deliver it to every watcher whose scope matches. Serialization and encoding dominate, and it is per watcher — a hundred controllers watching Pods means a hundred deliveries per pod change. High-churn objects therefore have super-linear blast radius: a controller that writes status on every reconcile multiplies its own load onto every other watcher of that type. **Client memory.** An informer holds the full object for everything in its scope. On a large cluster, all Pods can be gigabytes across processes; Secrets are worse per byte and more sensitive. Memory pressure in controllers then causes slow event draining, which increases lag and makes the cache staler — a feedback loop. **Recovery.** LIST is the expensive operation: it materialises and serializes a whole collection. It is fine occasionally, catastrophic in a herd. Anything that causes many clients to relist at once — an API server rollout, a network partition healing, aged-out resume cursors — is the real scaling event to design against. ## Client-side levers 1. **Share, don't duplicate.** A SharedInformerFactory gives one watch and one cache per resource type per process. Multiple independent informers for the same type inside one binary is a common and avoidable multiplier, especially when several controllers are bundled into one manager. 2. **Narrow the scope.** Namespace-scoped informers, label selectors, and field selectors cut both delivery and memory. The canonical example is the kubelet watching only pods with `spec.nodeName` equal to its own node — without that, every node would receive every pod change in the cluster. Bookmarks make narrow selectors safe by keeping the resume cursor fresh despite low event volume. 3. **Trim objects before caching.** Transform functions applied on the way into the store can drop `managedFields`, last-applied annotations and other bulk that controllers never read, cutting memory materially with no behaviour change. 4. **Right-size resync.** Resync replays the cache through handlers; it adds reconcile work without refreshing anything from the server. Short intervals multiply CPU across every object for no consistency benefit. Prefer long intervals or none, and rely on correct event handling plus explicit requeues. 5. **Reduce write amplification.** Controllers should write only when the computed state differs from what is stored — no unconditional status updates each pass, no timestamp fields that change every reconcile. Every avoided write is a delivery avoided at every watcher. 6. **Keep high-frequency data out of the API.** Metrics, heartbeats and per-request state do not belong in objects everyone watches. Lease objects exist precisely to keep node heartbeats small and separate from the Node object; the general principle is to isolate churn into small, narrowly watched resources. 7. **Handle failures gracefully.** Exponential backoff with jitter on reconnect, bounded retry, and correct 410 handling turn a potential stampede into a spread-out recovery. ## The judgement part The interesting decisions are architectural. Do you build one controller with a broad watch or several with narrow ones? Should a platform component watch all namespaces, or run per-tenant with scoped caches? Is a new custom resource's status going to be written every few seconds by design, and who will pay for that? Should this data live in the Kubernetes API at all, or in a purpose-built store, given that the API is a coordination substrate rather than a high-throughput datastore? A strong answer names the cost model explicitly (objects times watchers times churn, plus memory, plus recovery), picks the levers with the highest ratio of saving to complexity, and treats control-plane capacity as a shared resource that individual controllers can and do degrade for everyone.