Kubernetes defines the metrics.k8s.io, custom.metrics.k8s.io and external.metrics.k8s.io APIs; what serves each one, and where does kube-state-metrics fit?

middleshould knowfreq 46%

answer

one name, one backend
resource vs object vs outside
name equals version dot group
adapter translates, backend stores
state exporter, no APIService

basics

~20 s

Each group is served by a separate server registered through an APIService: metrics-server for metrics.k8s.io, and adapters for custom and external metrics. kube-state-metrics is not an API at all but a plain exporter of object state.

solid answer

~40 s

All three are aggregated APIs: kube-apiserver does not implement them; an APIService named <version>.<group> points each group-version at a Service. metrics.k8s.io carries CPU and memory usage and is served by metrics-server. custom.metrics.k8s.io carries metrics about Kubernetes objects, such as requests per second per pod, and is served by an adapter that translates the call into a query against a metrics backend. external.metrics.k8s.io carries metrics not tied to any object, such as a queue's depth, and is served by an adapter such as KEDA's metrics server. Because the APIService name is the group-version, only one backend can serve each group-version per cluster. kube-state-metrics registers no APIService: it exposes object state on a plain /metrics endpoint for a metrics backend to scrape.

All three are **aggregated APIs**: kube-apiserver does not implement them; an `APIService` named `<version>.<group>` points each group-version at a Service. `metrics.k8s.io` carries CPU and memory usage and is served by metrics-server. `custom.metrics.k8s.io` carries metrics *about Kubernetes objects*, such as requests per second per pod, and is served by an adapter that translates the call into a query against a metrics backend. `external.metrics.k8s.io` carries metrics *not tied to any object*, such as a queue's depth, and is served by an adapter such as KEDA's metrics server. Because the APIService name is the group-version, only one backend can serve each group-version per cluster. kube-state-metrics registers no APIService: it exposes object state on a plain `/metrics` endpoint for a metrics backend to scrape.

code

yaml · 12 lines

yaml

apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  name: v1beta1.external.metrics.k8s.io
spec:
  group: external.metrics.k8s.io
  version: v1beta1
  service:
    name: keda-metrics-apiserver
    namespace: keda
  groupPriorityMinimum: 100
  versionPriority: 100

go deeper

for a junior

Know that three metrics API groups exist, that metrics-server owns the CPU and memory one, and that kube-state-metrics is a separate exporter.

for a middle

Explain the APIService fields, the name rule, and the difference between object-attached custom metrics and object-less external metrics.

for a senior

Anticipate the one-backend-per-group-version collision when two autoscaling products are installed, and design one owner for each metrics group.

for a principal

Decide which metrics sources autoscaling may depend on, weighing an adapter's extra hop and failure mode against the value of scaling on business signals.

The shared mechanism: an APIService per group-version

Kubernetes can serve an API group from a server other than kube-apiserver. The aggregation layer inside kube-apiserver keeps a table built from APIService objects (apiregistration.k8s.io/v1). Each one says: "requests for this group and version go to this Service." Key fields:

spec.group and spec.version — the group-version being delegated
spec.service — namespace, name and optional port of the backing Service
spec.caBundle or spec.insecureSkipTLSVerify — how kube-apiserver trusts that server's TLS certificate
spec.groupPriorityMinimum and spec.versionPriority — ordering in discovery output
status.conditions — an Available condition with a reason such as Passed, MissingEndpoints or FailedDiscoveryCheck

Validation requires the object's name to equal spec.version + "." + spec.group, for example v1beta1.metrics.k8s.io. That single rule has an important consequence: there is exactly one backend per group-version in a cluster.

The three metrics APIs

API group	What it carries	Typical server	Scope of a metric
`metrics.k8s.io`	CPU and memory usage	metrics-server	Nodes, pods, containers
`custom.metrics.k8s.io`	Application metrics attached to a Kubernetes object	An adapter in front of a metrics backend (prometheus-adapter is common)	A described object: a pod, a Service, a Deployment
`external.metrics.k8s.io`	Metrics about things outside the cluster's object model	An adapter such as KEDA's metrics server	Namespace plus a metric name and label selector

Resource metrics are the fixed, built-in pair. metrics-server scrapes kubelets and answers from memory.

Custom metrics answer "what is metric X for object Y?" — for example, requests per second for each pod of a ticket-booking checkout. The adapter holds rules that map a Kubernetes object to a backend series and runs the query on demand. The adapter is a translator; the numbers are stored elsewhere.

External metrics answer "what is metric X?" when no Kubernetes object owns it — the number of unconfirmed bookings waiting in a message queue outside the cluster. KEDA registers an APIService named v1beta1.external.metrics.k8s.io pointing at its keda-metrics-apiserver Service.

The one-backend-per-group-version consequence

Because the name is fixed by the group-version, two products that both want to serve external metrics collide:

Adapter A applies v1beta1.external.metrics.k8s.io pointing at its Service.
Adapter B applies an object with the same name pointing at its Service.
There is still one object. Whichever write landed last decides where every external-metrics request goes, and the other adapter silently stops being consulted.

The fix is a design decision, not a flag: pick one component to own each group-version, and have it front every source you need.

How a client discovers these groups

An aggregated group looks identical to a built-in one from the outside. kubectl api-resources lists nodes and pods under metrics.k8s.io exactly like any other resource, and RBAC applies normally: a user needs get or list on pods in the metrics.k8s.io group to run kubectl top pods. What differs is only where the request ends up. kube-apiserver still authenticates and authorises the caller, then forwards the request over TLS to the backing Service, passing the caller's identity in request headers that the extension server trusts because they arrive with the aggregator's front-proxy client certificate. This is why an adapter is a real API server with its own availability, not a plugin loaded into kube-apiserver.

Where kube-state-metrics fits

kube-state-metrics is frequently confused with metrics-server, but it is a different kind of component:

It watches Kubernetes objects and turns their state into metrics — kube_pod_status_phase, kube_deployment_status_replicas_available and many more.
It exposes them on a plain-text /metrics HTTP endpoint (port 8080 by default) for a metrics backend to scrape.
It registers no APIService and serves nothing under /apis.
It reports state ("how many replicas are available"), never usage ("how much CPU").

So a question like "how many of the 1,180 pods in the checkout namespace are Pending?" is a kube-state-metrics series in a metrics backend. A Kubernetes consumer can only see such a value through the aggregation layer if a custom or external metrics adapter translates it.

Summary of the boundaries

Aggregated metrics APIs are read paths into other servers, not storage inside kube-apiserver or etcd.
An unavailable backing server makes that group fail with 503 and shows up as Available=False on its APIService.
Exporters like kube-state-metrics sit outside the Kubernetes API entirely.

## The shared mechanism: an APIService per group-version Kubernetes can serve an API group from a server other than kube-apiserver. The **aggregation layer** inside kube-apiserver keeps a table built from **`APIService`** objects (`apiregistration.k8s.io/v1`). Each one says: "requests for this group and version go to this Service." Key fields: - `spec.group` and `spec.version` — the group-version being delegated - `spec.service` — `namespace`, `name` and optional `port` of the backing Service - `spec.caBundle` or `spec.insecureSkipTLSVerify` — how kube-apiserver trusts that server's TLS certificate - `spec.groupPriorityMinimum` and `spec.versionPriority` — ordering in discovery output - `status.conditions` — an `Available` condition with a reason such as `Passed`, `MissingEndpoints` or `FailedDiscoveryCheck` Validation requires the object's name to equal `spec.version + "." + spec.group`, for example `v1beta1.metrics.k8s.io`. That single rule has an important consequence: **there is exactly one backend per group-version in a cluster**. ## The three metrics APIs | API group | What it carries | Typical server | Scope of a metric | |---|---|---|---| | `metrics.k8s.io` | CPU and memory usage | metrics-server | Nodes, pods, containers | | `custom.metrics.k8s.io` | Application metrics attached to a Kubernetes object | An adapter in front of a metrics backend (prometheus-adapter is common) | A described object: a pod, a Service, a Deployment | | `external.metrics.k8s.io` | Metrics about things outside the cluster's object model | An adapter such as KEDA's metrics server | Namespace plus a metric name and label selector | **Resource metrics** are the fixed, built-in pair. metrics-server scrapes kubelets and answers from memory. **Custom metrics** answer "what is metric X for object Y?" — for example, requests per second for each pod of a ticket-booking checkout. The adapter holds rules that map a Kubernetes object to a backend series and runs the query on demand. The adapter is a translator; the numbers are stored elsewhere. **External metrics** answer "what is metric X?" when no Kubernetes object owns it — the number of unconfirmed bookings waiting in a message queue outside the cluster. KEDA registers an `APIService` named `v1beta1.external.metrics.k8s.io` pointing at its `keda-metrics-apiserver` Service. ## The one-backend-per-group-version consequence Because the name is fixed by the group-version, two products that both want to serve external metrics collide: 1. Adapter A applies `v1beta1.external.metrics.k8s.io` pointing at its Service. 2. Adapter B applies an object with the same name pointing at *its* Service. 3. There is still one object. Whichever write landed last decides where every external-metrics request goes, and the other adapter silently stops being consulted. The fix is a design decision, not a flag: pick one component to own each group-version, and have it front every source you need. ## How a client discovers these groups An aggregated group looks identical to a built-in one from the outside. `kubectl api-resources` lists `nodes` and `pods` under `metrics.k8s.io` exactly like any other resource, and RBAC applies normally: a user needs `get` or `list` on `pods` in the `metrics.k8s.io` group to run `kubectl top pods`. What differs is only where the request ends up. kube-apiserver still authenticates and authorises the caller, then forwards the request over TLS to the backing Service, passing the caller's identity in request headers that the extension server trusts because they arrive with the aggregator's front-proxy client certificate. This is why an adapter is a real API server with its own availability, not a plugin loaded into kube-apiserver. ## Where kube-state-metrics fits **kube-state-metrics** is frequently confused with metrics-server, but it is a different kind of component: - It **watches Kubernetes objects** and turns their state into metrics — `kube_pod_status_phase`, `kube_deployment_status_replicas_available` and many more. - It exposes them on a plain-text `/metrics` HTTP endpoint (port 8080 by default) for a metrics backend to **scrape**. - It registers **no APIService** and serves nothing under `/apis`. - It reports *state* ("how many replicas are available"), never *usage* ("how much CPU"). So a question like "how many of the 1,180 pods in the checkout namespace are Pending?" is a kube-state-metrics series in a metrics backend. A Kubernetes consumer can only see such a value through the aggregation layer if a custom or external metrics adapter translates it. ## Summary of the boundaries - Aggregated metrics APIs are **read paths into other servers**, not storage inside kube-apiserver or etcd. - An unavailable backing server makes that group fail with 503 and shows up as `Available=False` on its APIService. - Exporters like kube-state-metrics sit **outside** the Kubernetes API entirely.

Your team wants KEDA and a second product that also serves external metrics in the same cluster. What do you tell them?
Only one APIService can exist for `v1beta1.external.metrics.k8s.io`, so only one of them can answer external-metrics requests; whichever install wrote the object last wins and the other is silently bypassed. Choose one owner of the group, and route the other product's data through it, for example by having KEDA query the same metrics backend through one of its scalers.
Does creating an APIService store any metric data in etcd?
No. Only the APIService object itself is stored. Every request under that group-version is proxied live to the backing Service, which answers from its own memory or by querying its own backend. If that server is down, there is nothing cached in kube-apiserver to fall back on.

saying these in an interview costs you the question

kube-state-metrics serves the metrics.k8s.io API
custom metrics are stored in etcd by kube-apiserver
two adapters can both serve external.metrics.k8s.io side by side
external metrics must be attached to a Pod object
an APIService can be given any name you like

Kubernetes defines the metrics.k8s.io, custom.metrics.k8s.io and external.metrics.k8s.io APIs; what serves each one, and where does kube-state-metrics fit?

§ answer

basics

solid answer

code

§ go deeper

▾level notes4 levels

▾deep dive~4 min

The shared mechanism: an APIService per group-version

The three metrics APIs

The one-backend-per-group-version consequence

How a client discovers these groups

Where kube-state-metrics fits

Summary of the boundaries

▾common follow-ups2 questions

▾red flags5 flags

The shared mechanism: an APIService per group-version

The three metrics APIs

The one-backend-per-group-version consequence

How a client discovers these groups

Where kube-state-metrics fits

Summary of the boundaries

answer

go deeper