Walk through what happens to a request such as 'kubectl apply -f pod.yaml' inside the Kubernetes API server, from the TLS connection to the object being stored in etcd.
answer
- authn → authz → admission → persist
- authn yields username + groups only; no User object
- authz sees verb/resource, never object fields
- mutating first, then schema validation, then validating
- 401 = authn, 403 = authz, named-policy error = admission
basics
~20 sThe API server terminates TLS, then runs three gates in order: authentication (who are you — cert, service-account token, or OIDC token), authorization (may this identity do this verb on this resource — usually RBAC), and admission (mutating plugins/webhooks may edit the object, then validating ones may reject it). Only then is the object schema-validated and written to etcd.
solid answer
~60 s1. **Transport** — the client connects over TLS to kube-apiserver; the request is an HTTP call like `POST /api/v1/namespaces/default/pods`. 2. **Authentication** — a chain of authenticators tries to establish an identity: client certificate, service-account bearer token, OIDC ID token, or an authentication webhook. First success wins and yields a username plus groups. All fail ⇒ the request becomes `system:anonymous` or gets 401. 3. **Authorization** — the identity plus the request attributes (verb, resource, namespace, name) go to the authorization modules, normally RBAC and Node. Any module returning allow short-circuits to allow; if none allows, the answer is 403. 4. **Admission** — **mutating** admission plugins and webhooks run first and may patch the object (defaults, sidecar injection); then the object is schema-validated; then **validating** plugins and webhooks run and may only accept or reject. ResourceQuota is evaluated here too. 5. **Persistence** — the accepted object is serialised and written to etcd; watchers (schedulers, controllers) see it and act. Mnemonic: **authn → authz → admission → persist**, and admission is the only stage that can change the object.
code
bash · 12 lines# Stage 1 — what identity does the API server see for my credentials?
kubectl auth whoami
# Stage 2 — would authorization allow this verb on this resource?
kubectl auth can-i create pods --namespace default
kubectl auth can-i list secrets --namespace kube-system --as [email protected]
# Stage 3 — run mutating + validating admission without persisting
kubectl apply -f pod.yaml --dry-run=server
# Stage 4 — the write happened only if the object is now readable
kubectl get pod -n default my-pod -o yamlgo deeper
Recite the four stages in order and what each decides, and map 401 to authentication and 403 to authorization.
Add the mechanics: authenticator chain with first-success-wins, RBAC being additive and field-blind, mutating-then-validating with schema validation in between.
Explain why the ordering forces field-level policy into admission, how ResourceQuota and dry-run behave, and how you use the stage boundaries to triage a failing request quickly.
Use the pipeline as the frame for cluster security architecture: which control belongs at which stage, the cost each stage adds to every write, and what a compromise of each layer would mean.
## One door into the cluster Everything in Kubernetes goes through kube-apiserver: `kubectl`, controllers, the scheduler, kubelets, and every operator. There is no side channel that writes to etcd directly. That is why the API-server request pipeline is *the* security model of Kubernetes — understanding its four stages tells you where every control lives. ## Stage 0 — transport The API server serves HTTPS only. If the client presents a certificate during the TLS handshake, it is available to the certificate authenticator later; otherwise the connection is still established and identity must come from a header. The REST path itself encodes the target: `POST /api/v1/namespaces/default/pods` (core group) or `/apis/apps/v1/namespaces/default/deployments` (named group). The HTTP method maps to a verb — POST→create, GET→get/list, PUT→update, PATCH→patch, DELETE→delete, plus watch for streaming reads. ## Stage 1 — authentication (who are you?) The API server runs an ordered chain of **authenticators**; each either returns an identity or abstains, and the first success wins: - **X.509 client certificate** signed by a CA the API server trusts: the certificate's Common Name becomes the username, its Organization values become groups. - **Service-account bearer token**: a signed JWT in `Authorization: Bearer …`, yielding `system:serviceaccount:<namespace>:<name>`. - **OIDC ID token** from an external identity provider, mapping configured claims to username and groups — the usual mechanism for humans. - **Authentication webhook**, used by managed clouds to validate their own token formats. An authenticated request carries only a **username, UID, and a list of groups** — Kubernetes stores no User object. If every authenticator abstains, the request is either rejected with 401 or, when anonymous access is enabled, continues as `system:anonymous` in the group `system:unauthenticated`. ## Stage 2 — authorization (may you do it?) The request is reduced to attributes: user, groups, verb, API group, resource, subresource, namespace, and object name (or, for non-resource endpoints such as `/healthz`, just the path). These go to the configured authorization modules, typically **Node** (restricting each kubelet to objects relevant to its own node) and **RBAC** (Roles/ClusterRoles bound to subjects), sometimes a **Webhook** for an external decision service. The modules form a chain: a module may answer allow, deny, or no-opinion; the first explicit answer decides, and if every module abstains the request is denied with 403. RBAC is purely additive — it grants, it never denies — so a 403 usually means "nothing granted it", not "something forbade it". Note the ordering consequence: **authorization runs before the object body matters**. You cannot express "may create pods, but only non-privileged ones" in RBAC, because RBAC sees the verb and resource, not the fields. That gap is exactly what the next stage exists for. ## Stage 3 — admission (should this specific object exist?) Admission controllers see the whole object and run in two sub-phases: 1. **Mutating** — built-in plugins (e.g. `DefaultStorageClass`, `ServiceAccount`) and registered mutating webhooks, which may patch the object: fill defaults, inject a sidecar or environment variables, rewrite an image to a digest. 2. **Object schema validation** — the mutated object is checked against the API schema, so a webhook cannot produce an invalid object. 3. **Validating** — built-in plugins, ValidatingAdmissionPolicies (CEL, in-process), and validating webhooks. These may only accept or reject; they cannot modify. `ResourceQuota` is evaluated late here, since it must count what will actually be stored. A rejection at this stage produces a 4xx with the controller's message — which is why admission errors read very differently from RBAC's terse "forbidden". ## Stage 4 — persistence The accepted object is versioned, optionally encrypted at rest, and written to etcd through the storage layer. Only now does the object exist. Everything after that is asynchronous: the scheduler watches for unscheduled pods and binds one to a node, the kubelet on that node watches for pods bound to it and starts containers. None of that is part of the request; `kubectl apply` returns as soon as the object is persisted, which is why a successful apply says nothing about whether the workload actually runs. ## Why the order is the whole point Each stage answers a strictly different question and can only be reached by passing the previous one: *who are you* (identity), *are you allowed to perform this operation on this kind of thing* (coarse, field-blind), *is this particular object acceptable* (fine-grained, field-aware). Security reviews follow the same order, and so does debugging: a 401 is authentication, a 403 is authorization/RBAC, and a verbose message naming a webhook or policy is admission.
- You get 'Error from server (Forbidden)'. Which stage failed, and how do you confirm it?Forbidden is HTTP 403, which is the authorization stage — you were authenticated successfully but no authorization module granted the verb/resource combination. Confirm with kubectl auth whoami to see the identity the API server assigned, then kubectl auth can-i <verb> <resource> to reproduce the decision. If instead the message names a webhook or policy and describes the object's contents, the request passed authorization and was rejected at admission.
- Why can't RBAC express 'this user may create pods, but not privileged ones'?Authorization runs before the object's fields are considered: the decision inputs are user, groups, verb, API group, resource, subresource, namespace and name — the request body is not among them. Field-level conditions therefore have to be enforced at admission, by Pod Security Standards, a ValidatingAdmissionPolicy, or a policy engine. This split is deliberate; it keeps the authorization layer cheap and cacheable.
An airport: passport control proves who you are, the boarding pass check says whether you may take this flight, and security screening inspects what you are actually carrying — and only screening can make you repack your bag.
saying these in an interview costs you the question
- Putting admission before authorization, or claiming admission decides who you are.
- Believing Kubernetes stores user accounts as objects in etcd.
- Saying validating webhooks can modify the object — only mutating ones can.
- Thinking a successful kubectl apply means the pod is running; it only means the object was persisted.
- Confusing 401 (authentication failed) with 403 (authenticated but not authorized).