Aggregated APIs & Metrics

## What metrics-server is **metrics-server** is a Kubernetes SIG-maintained add-on, usually running as a Deployment in `kube-system`, that implements the **resource metrics API**. That API lives in the API group `metrics.k8s.io` and exposes two read-only resources: - `nodes` — current CPU and memory usage per Node - `pods` — current CPU and memory usage per Pod, broken down by container It exists so that core Kubernetes features have a *standard, always-present* answer to one narrow question: "how much CPU and memory is this thing using right now?" The two main consumers are `kubectl top` and the **HorizontalPodAutoscaler** when it targets CPU or memory utilisation. ## How the data flows 1. Every kubelet already measures container usage through the container runtime and publishes it on its HTTPS port (10250 by default) at `/metrics/resource`. 2. metrics-server scrapes that endpoint on every node at a fixed interval, set by `--metric-resolution`. The upstream manifest sets `15s`; the flag rejects anything below `10s`. 3. It keeps the **last two scrapes** per container and node in memory. CPU usage is a rate, so it is computed from the difference between those two points; memory is the latest working-set value. 4. It serves the result as an ordinary-looking Kubernetes API. An `APIService` object named `v1beta1.metrics.k8s.io` tells kube-apiserver to forward every request under `/apis/metrics.k8s.io/v1beta1/` to the `metrics-server` Service. 5. `kubectl top pods` and the HPA controller call kube-apiserver as usual; the **aggregation layer** proxies the call to metrics-server, which answers from memory. Because the answer comes from memory, a request for a 1,180-pod namespace in a ticket-booking checkout is cheap — no query engine, no disk. ## What it deliberately is not The project README states its non-goals plainly: it is not for non-Kubernetes clusters, not "an accurate source of resource usage metrics", and not for autoscaling on anything other than CPU and memory. | Expectation | Reality in metrics-server | |---|---| | History / trends | None — only the latest window; a restart starts from empty | | Application metrics (requests per second, queue depth) | Not served; those come from custom or external metrics adapters | | Dashboards and alerting | Not provided; there is no query language and no rules | | Export to a backend | Nothing is pushed anywhere | | Billing-grade accuracy | Explicitly disclaimed; it is sized for autoscaling decisions | | Object state (desired vs ready replicas) | Not its job; that is kube-state-metrics | A metrics backend such as Prometheus is a **separate, parallel pipeline**. It scrapes its own targets and stores time series; it does not use metrics-server, and metrics-server does not feed it. ## Reading `kubectl top` correctly - The numbers are **recent usage**, not requests or limits. A pod requesting `500m` but idle shows a few millicores. - Right after a pod starts, or right after metrics-server restarts, a pod may be missing: two scrapes are needed before a CPU rate exists. - `kubectl top pod --containers` shows the per-container split; `--sort-by=cpu` or `--sort-by=memory` orders the list. - `kubectl top node` values come from the kubelet's node-level measurement, so they include system daemons and do not equal the sum of the pods. ## Operating it on a small cluster On a 5-node edge cluster in a retail store, metrics-server is often the *only* metrics component, because shipping a full monitoring stack to every store is expensive. That is fine for `kubectl top` and CPU-based scaling of the checkout, but the team must accept that nobody can answer "what was checkout's memory at 14:05 yesterday?" from it. Its footprint grows with the number of pods and nodes it tracks, so its memory request is sized to cluster scale rather than traffic. It also depends on the aggregation layer being configured on kube-apiserver and on kubelets presenting serving certificates it can verify; when either is broken, `kubectl top` fails even though every workload is healthy. ## Common interview traps - **"A metrics backend makes metrics-server redundant"** — they coexist; removing metrics-server breaks `kubectl top` and resource-based HPAs even when a metrics backend is healthy. - **"It reads cAdvisor directly"** — current releases scrape the kubelet's `/metrics/resource` endpoint; the kubelet is the single source. - **"More replicas give more history"** — running several metrics-server replicas is for availability of the API, not for retention; each replica still holds only the latest window. - **"`kubectl top` is authoritative for sizing requests"** — it is a point-in-time glance; sizing needs percentiles over days, which only a stored time series can give.