Cluster Architecture

An APIService registers a group/version and points at a Service; kube-apiserver then proxies every request for that group/version to that extension server, which implements storage and semantics itself. Choose it when you need non-etcd storage or custom verbs.

A watch streams change events occurring after a given resourceVersion, which the server can only replay from a bounded in-memory window. Once that version ages out, the server returns 410 and the client must re-list to get a fresh resourceVersion and re-watch.

Because of leader election: all three instances start, but each competes to hold a Lease object in the API server. Only the lease holder runs its control loops; the others idle as hot standbys and take over if the lease expires.

The work queue deduplicates keys, guarantees one worker per key at a time, and requeues failures with exponential backoff. A tight retry loop would block other objects, hammer the API server, and turn one broken object into a control-plane load problem.

etcd keeps every historical revision and enforces a space quota (2 GiB by default). Compaction discards revisions older than a chosen point, freeing space inside the file; defragmentation rewrites the file so freed pages return to the filesystem. Afterwards you must disarm the NOSPACE alarm before writes resume.

Two writers each treat that field as theirs and keep correcting each other's drift. Find them via metadata.managedFields, the audit log and controller logs, then give the field exactly one owner — usually by removing it from the declarative manifest and letting the autoscaler or operator own it.

An informer runs one list-watch per resource type, keeps the objects in an in-memory indexed store, and fires handlers on changes; shared means many controllers in the process reuse that one stream and cache. Cache reads are fast and free but eventually consistent, so they can be stale and must never be mutated.

Pick the member count from the failures you must survive: three tolerate one loss, and five tolerate two, including one planned. Spread members so no single rack or power domain holds a majority, which needs at least three failure domains with low-latency links.

That delay comes from client-go's own token-bucket rate limiter (5 QPS, burst 10 unless you change them), which holds requests before they leave the process. An HTTP 429 means the API server's Priority and Fairness filter rejected a request that did arrive.

A standby takes over only after the leader's Lease looks expired: 15 seconds by default, plus a jittered 2-second retry. The rest of a 47-second stall usually comes from endpoint failover, stuck connections and cache sync.

A BOOKMARK is a periodic watch event carrying only an up-to-date resourceVersion and no object change. It lets an idle client advance its resume cursor, so after a disconnect it can resume instead of getting 410 Gone and re-listing everything.

Classify the load: expensive unpaginated LISTs dominate memory, watches dominate steady state. Add replicas for throughput, then constrain clients with pagination, selectors and informers, and enforce fairness with API Priority and Fairness so one client cannot starve the rest.

Cost scales with objects times watchers times change rate, plus the memory each client caches. Reduce it by sharing informers per process, scoping watches with label and field selectors, trimming cached objects, avoiding short resyncs and relist storms, and keeping fast-churning data out of watched objects.