Where do the numbers reported by kubectl top pods and kubectl top nodes come from, and what should you avoid concluding from them?

## The data path `kubectl top pod` and `kubectl top node` query the aggregated API group `metrics.k8s.io`. That API is served by **metrics-server**, an optional cluster add-on. metrics-server discovers nodes, scrapes each kubelet's resource metrics endpoint, and stores **only the latest sample per object in memory**. Nothing is persisted; restarting metrics-server empties it. The same API backs the Horizontal Pod Autoscaler, which is why a missing or unhealthy metrics-server breaks autoscaling and `kubectl top` at the same time — the tell is an error that the Metrics API is not available. The kubelet's numbers come from cAdvisor reading cgroup accounting for each container, so they are the kernel's view of the container's resource use, not the application's self-report. ## What the two numbers actually mean **CPU** is reported in cores or millicores and is a **rate averaged over the scrape window**, computed from the cumulative CPU-time counter. Two consequences: short spikes are smoothed away, and a freshly started pod may show nothing until two samples exist. If a pod is being CPU-throttled by its limit, `top` shows usage pinned near the limit but does not show the throttling itself — that lives in container throttling counters, which is monitoring-stack territory. **Memory** is working-set bytes: memory charged to the cgroup minus inactive file-backed pages. It therefore includes active page cache from files the container read, and is neither RSS nor the language runtime's heap. Practical reading: a JVM with a 512Mi heap can show 700Mi or more of working set once metaspace, thread stacks, direct buffers and cache are counted, and that is normal, not a leak. Working set is close to what the kernel considers when deciding to OOM-kill a container against its memory limit, which makes it the right number for "am I near the limit" and the wrong number for "how big is my heap". ## Useful invocations - `kubectl top pod -n prod --sort-by=cpu` — rank the noisiest pods. - `kubectl top pod mypod --containers` — split by container, identifying whether the sidecar or the app is consuming. - `kubectl top node` — per-node CPU and memory usage plus the percentage of **allocatable** (not capacity: allocatable subtracts kubelet and system reservations). - `kubectl top pod -A --sort-by=memory` — quick cluster-wide offenders. ## The comparison people get wrong `kubectl top node` reports **actual usage**. `kubectl describe node` reports **requests and limits** of the pods scheduled there. These answer different questions: - Pods are Pending and unschedulable while `top node` shows 15% usage: the node is out of **requested** capacity, not out of real capacity. The fix is right-sizing requests, not adding CPU. - `top node` shows 95% memory while requests total 40%: workloads use far more than they request, and the node is at genuine risk of pressure and eviction. Stating which of the two you are looking at is a strong signal of understanding. ## The limits to declare No history, no percentiles, no per-container time series, one sample per object, and a delay of up to a scrape interval or two. That makes `kubectl top` a **spot check**, appropriate for "what is hot right now" during an incident, and unfit for trend analysis, leak confirmation, capacity planning, or alerting — all of which belong to the monitoring stack with real retention. A candidate who tries to prove a memory leak from two `kubectl top` readings is over-reading the tool; the honest use is to spot the outlier pod and then go look at its retained metrics or take a heap dump.