Kubernetes lets you add API endpoints either with a CustomResourceDefinition or through the API aggregation layer and an APIService object. Explain how aggregation works inside kube-apiserver and when you would choose it over a CRD.

## Two extension mechanisms, one API surface Both mechanisms make `kubectl get <thing>` work and both are governed by ordinary RBAC, but the machinery behind them is completely different. **CustomResourceDefinition.** You submit a CRD describing a group, versions, kind, scope and an OpenAPI v3 structural schema. kube-apiserver dynamically creates handlers, stores objects in the same etcd it uses for built-ins, validates against your schema (with CEL validation rules if you supply them), and supports watch, the `/status` and `/scale` subresources, printer columns, and server-side apply. You run no additional server; typically you also run a controller, but the controller is just another API client. **Aggregation.** You run a second API server — one built on the same `k8s.io/apiserver` library the core uses — expose it as a Service, and register it: ```yaml apiVersion: apiregistration.k8s.io/v1 kind: APIService metadata: name: v1beta1.metrics.k8s.io spec: group: metrics.k8s.io version: v1beta1 service: name: metrics-server namespace: kube-system port: 443 groupPriorityMinimum: 100 versionPriority: 100 ``` kube-apiserver contains an **aggregator** in front of its own handlers. It merges the registered group/version into discovery, and any request whose path begins `/apis/metrics.k8s.io/v1beta1/...` is reverse-proxied to that Service rather than handled locally. ## The trust chain Proxying raises an obvious question: how does the extension server know who the original caller was, and how does it authorise them? - **Front-proxy identity.** kube-apiserver authenticates itself to the extension server with a client certificate signed by the requestheader CA, and forwards the original user's identity in headers (`X-Remote-User`, `X-Remote-Group`, `X-Remote-Extra-*`). The extension server trusts those headers *only* from a client whose certificate chains to the CA published in the `extension-apiserver-authentication` ConfigMap in `kube-system`. Without that pinning, anyone able to reach the extension server could forge an admin identity — this is the single most important thing to get right when writing one. - **Delegated authentication and authorisation.** The extension server does not implement RBAC. It calls back into the main API server with `TokenReview` (to validate bearer tokens presented directly) and `SubjectAccessReview` (to ask "may this user do this verb on this resource?"). So cluster RBAC still governs the aggregated API, which is why `kubectl auth can-i get pods.metrics.k8s.io` behaves normally. ## Storage freedom is the real differentiator A CRD's objects always land in etcd. That is fine for configuration-shaped data — tens to thousands of objects, changing occasionally. It is disastrous for high-frequency or high-volume data, because etcd is a consensus store with a practical database-size limit and every write costs a Raft round trip. metrics-server is the canonical aggregated API precisely for this reason: resource usage samples for every Pod, refreshed constantly, are held in memory with a short rolling window and never persisted. `kubectl top` and the HorizontalPodAutoscaler read them through `metrics.k8s.io` exactly as if they were normal objects. The same logic drives `custom.metrics.k8s.io` and `external.metrics.k8s.io` adapters that front Prometheus or a cloud metrics service. Other reasons to aggregate: verbs or endpoint shapes a CRD cannot express (streaming, proxying, non-CRUD actions, an arbitrary subresource with custom semantics), a need to expose an existing system's data as Kubernetes objects without copying it, or per-request logic that admission webhooks cannot express. ## The cost, and why CRDs are the default Aggregation introduces a control-plane dependency. The aggregator health-checks each APIService, and an unavailable one is reported as `Available=False` on the APIService object. The user-visible consequence is disproportionate: discovery aggregates every registered group, so a hung extension API makes `kubectl` commands slow or produce `error: unable to retrieve the complete list of server APIs`, even for people who never use that API. `kubectl get pods` becoming slow because metrics-server is unhealthy is a well-known and initially baffling symptom. You also now own: TLS certificates and their rotation, delegated authn/authz wiring, HA for the extension server, version skew against the core API, and correct RBAC on the `extension-apiserver-authentication` ConfigMap. By contrast a CRD has essentially no availability surface of its own. The main exception is a **conversion webhook**, which reintroduces a dependency on the same footing as an aggregated server. ## How to decide Start with a CRD. Move to aggregation only when you can name a requirement a CRD structurally cannot meet: data that must not live in etcd, volume or churn etcd cannot absorb, or verbs outside the CRUD-plus-subresource model. "We want it to feel native" is not a reason — CRDs already feel native. "We produce ten thousand samples a second" is. ## Inspecting what a cluster has ``` kubectl get apiservices # Local means core; others are aggregated kubectl get apiservice v1beta1.metrics.k8s.io -o yaml kubectl get crds ``` An APIService whose `SERVICE` column reads `Local` is served by kube-apiserver itself; anything naming a namespace/service is aggregated and is a live dependency of your control plane.