Cluster Architecture

kube-apiserver is the cluster's front door: a REST API that authenticates, authorises, validates and persists every object, and streams changes to watchers. Routing all writes through it gives one place for auth, validation, admission and audit.

kube-scheduler watches for Pods with an empty spec.nodeName, picks a suitable node, and writes that choice back to the API server (a binding). It does not start the container — the kubelet on the chosen node does that.

etcd is the cluster's only persistent database: a distributed, strongly consistent key-value store holding every API object (Pods, Deployments, Services, ConfigMaps, Secrets, RBAC, node state). Only kube-apiserver connects to it; everything else reads and writes through the API server.

Running Pods keep running: the kubelet keeps their containers alive and kube-proxy's installed Service rules keep routing. Anything that needs a write or fresh state stops working: scheduling, scaling, rollouts, rescheduling off failed nodes and kubectl.

The kubelet is the node agent. It watches the API server for Pods assigned to its node, tells the container runtime to create the containers described in each PodSpec, mounts volumes, runs probes, and reports Pod and node status back. It manages Pods only, not Deployments or ReplicaSets.

spec is the desired state you declare; status is the observed state the system reports. You and your tools write spec, controllers write status. Kubernetes keeps acting until status matches spec.

Alpha versions are off by default and may vanish in any release; beta versions stay served at least 9 months or 3 releases after deprecation; GA versions are never removed within Kubernetes v1. New betas are off by default since v1.24.

kube-apiserver checks each request against the FlowSchema objects, lowest matchingPrecedence first, and sends it to the PriorityLevelConfiguration the matching schema names. If that level has no free seats, the request waits in a fair queue by flow or gets HTTP 429.

TLS, then authentication (who), authorisation (may they), decode plus defaulting, mutating admission, schema validation, validating admission, then a write to etcd guarded by optimistic concurrency, followed by audit and watch fan-out.

It is one control-plane process hosting many independent controllers (Deployment, ReplicaSet, Node, Job, endpoints, service accounts…). Each watches its objects via the API server, compares desired spec to observed state, and makes one corrective API call — repeatedly, until they match.

etcd uses Raft: a write must be replicated to a majority (quorum) before it commits, so an N-member cluster tolerates (N-1)/2 failures. Even sizes add a member without adding fault tolerance. Without quorum etcd rejects writes, so the control plane freezes — but running Pods keep running.

Stacked etcd runs a member on each control-plane node beside kube-apiserver; external etcd uses dedicated hosts. Stacked needs fewer machines but couples failures: three stacked nodes tolerate one loss, and each loss removes an API server and an etcd member.

A controller (the ReplicaSet behind the Deployment) constantly compares desired replica count with the pods it actually sees, and creates one when the count is short. Level-triggered means it acts on current state, not on the delete event, so it recovers even if it missed the event.

Kubernetes updates use optimistic concurrency: your object carries the resourceVersion you read, and the server rejects the write if the live object has changed since. Handle it by re-reading the object fresh, reapplying your change, and retrying — or by sending a patch instead of a full update.

The client LISTs the objects once, notes the collection's resourceVersion, then opens a WATCH starting after that version and receives ADDED/MODIFIED/DELETED events as a stream. resourceVersion is an opaque cursor into the API server's change history, not a number to compare or interpret.

Back up with etcdctl snapshot save (v3 API, with client TLS certs), on a schedule, stored off-box. To restore: stop the API servers, stop etcd and move the old data directory aside, run etcdctl snapshot restore into a fresh directory, start etcd as a new single-member cluster, restart the control plane, then re-add members.

On startup the kubelet authenticates (usually a bootstrap token plus a CSR) and creates the Node object with its capacity and labels. It then renews a Lease in the kube-node-lease namespace every ~10s as a heartbeat and updates node conditions less often. NotReady means the kubelet stopped reporting or reported a problem — not that containers died.

Run kubectl api-resources to list each resource's group/version, kind, short name and scope, kubectl api-versions to list every served group/version, and kubectl explain to read the fields the cluster's published schema accepts.

Each group is served by a separate server registered through an APIService: metrics-server for metrics.k8s.io, and adapters for custom and external metrics. kube-state-metrics is not an API at all but a plain exporter of object state.

A full LIST makes kube-apiserver load, convert and serialize every matching object in one response, and Priority and Fairness charges it several seats. With limit, the server returns pages plus a continue token for the next request until the token comes back empty.

apiVersion is group/version and kind is the type; together they select a REST path like /apis/apps/v1/namespaces/ns/deployments. The server stores one storage version and converts to whichever served version a client requests.

kube-apiserver replicas are stateless and all active, so clients need one stable address that survives any single server's loss. Put a TCP pass-through load balancer or virtual IP in front of them, and have it health-check each server's /readyz endpoint.

CRI is the gRPC API the kubelet uses to talk to a container runtime over a local socket. The kubelet calls it to pull images and to create pod sandboxes and containers; containerd implements CRI natively and delegates actual container creation to a low-level OCI runtime such as runc.

kube-proxy runs on every node (usually as a DaemonSet) and programs the node's kernel so traffic to a Service's virtual IP is redirected to a healthy backend Pod. It is a control agent writing rules, not a data-path proxy: if it stops, existing rules keep working but new Services and endpoint changes stop being applied.

A static pod is defined by a manifest file in a directory the kubelet watches (typically /etc/kubernetes/manifests). The kubelet runs it directly, with no scheduler or controller involved. It creates a read-only mirror Pod in the API so you can see it, but deleting that mirror does not stop it — you must remove the file.

create fails if the object exists; replace overwrites the whole object, dropping fields you omitted; apply merges your manifest with the live object. Server-side apply moves that merge into the API server and records per-field owners in metadata.managedFields, so a conflicting write is rejected until you force it.

Prometheus scrapes targets directly, so it proves nothing about the aggregation path. Check the v1beta1.metrics.k8s.io APIService's Available condition and reason, then fix the failing hop: Service and endpoints, apiserver-to-pod reachability and TLS, or metrics-server-to-kubelet scraping.

Read the removal list for each release you cross, find live callers with kube-apiserver's apiserver_requested_deprecated_apis metric and k8s.io/deprecated audit annotations, scan Git for dormant manifests, then rewrite and verify with fresh audit events.

Confirm that the 429s come from Priority and Fairness by checking the X-Kubernetes-PF response headers and apiserver_flowcontrol_rejected_requests_total. Find the flow in the APF debug dumps, then move the operator into its own small priority level while its team fixes the call pattern.