Trace what kube-apiserver does with an HTTPS POST of a Deployment manifest, from the moment the request arrives until the object is durably stored. Name the stages in order and say what each can reject.

## Why the order is the interesting part Anyone can list the stages; interviews test whether you know why they are sequenced this way and what each stage can and cannot see. ## 1. Transport and authentication The connection terminates TLS at the API server. Authentication runs a chain of authenticators until one succeeds: X.509 client certificates (the CN becomes the username, Organization entries become groups — this is how kubeadm gives control-plane components identities), service-account bearer tokens (projected, audience-bound, time-limited JWTs), OIDC tokens, and authentication webhooks. Failure yields **401 Unauthorized**. Anonymous requests may be permitted and arrive as `system:anonymous` in group `system:unauthenticated`. The output is a username, a UID, groups, and extra attributes — nothing more. Authentication never decides what you may do. ## 2. Authorization Authorizers run in order and the first explicit allow or deny wins. Typically: the **Node** authorizer (constrains kubelets to objects relevant to their own node), **RBAC** (Roles/ClusterRoles bound to subjects), and optionally a **webhook** authorizer. The decision is made purely on request *attributes* — user, groups, verb, group/resource/subresource, namespace, name — **not on object contents**. That is the crucial limitation: RBAC cannot say "you may create Pods but not privileged ones", because it never sees the body. Content-based policy is admission's job. Failure yields **403 Forbidden**. ## 3. Decoding and defaulting The body is deserialized from JSON or YAML into the versioned external type, converted to the internal hub type, and **defaulted**: `spec.replicas` becomes 1 if omitted, `imagePullPolicy` is derived from the tag, the Service's `type` becomes `ClusterIP`, and so on. Because defaulting precedes admission, mutating webhooks observe the defaulted object, not the literal YAML the user submitted — a frequent source of confusion when a webhook "sees fields nobody wrote". Malformed input yields **400 Bad Request**. ## 4. Mutating admission Built-in mutating plugins run first (for example `ServiceAccount`, which attaches the default service account and its token projection, and `LimitRanger`, which fills in default resource requests). Then registered `MutatingAdmissionWebhook`s are called **serially**, in an order the API server chooses, each seeing the output of the previous one. This is where sidecar injectors, policy engines, and image-tag rewriters act. Each webhook is configured with `failurePolicy: Fail` or `Ignore`, a `timeoutSeconds`, and matching rules. `Fail` plus an unavailable webhook means writes to the matched resources stop cluster-wide — the classic self-inflicted outage, especially when the webhook's own Pods cannot be created because its webhook is down. Scoping rules narrowly and excluding critical namespaces is the mitigation. ## 5. Schema validation After all mutation, the object is validated: required fields, structural schema conformance, enum values, cross-field invariants implemented in Go. Running this *after* mutating admission is deliberate — a webhook must not be able to produce an invalid object. Failure yields **422 Unprocessable Entity** with a field-level `Status` describing what was wrong. ## 6. Validating admission The final gate: built-in validating plugins (`ResourceQuota`, `PodSecurity`), `ValidatingAdmissionWebhook`s (called **in parallel**, since none may mutate), and `ValidatingAdmissionPolicy` — CEL expressions evaluated in-process, which avoid the availability and latency cost of a webhook for the many policies that are just predicates over the object. Any rejection fails the request with **403** and the message the policy supplied. Because nothing may mutate here, the object that passes is exactly the object stored. ## 7. Persistence The storage layer encodes the internal object into the configured **storage version**, applies at-rest encryption for resource types configured for it (Secrets, typically), and writes to etcd under a key like `/registry/deployments/<ns>/<name>`. Writes are guarded by **optimistic concurrency**: an update carries the object's `resourceVersion`, and if the stored revision has moved on, the request fails with **409 Conflict** and the client must re-read and retry. This is why controller code is written as a read-modify-write retry loop, and why server-side apply exists as a conflict-aware alternative. ## 8. After the write The API server writes an **audit** event (at the stages the audit policy requests), returns the persisted object to the caller, and delivers the change to every open watch on that resource — which is how the deployment controller, the scheduler, and your dashboards learn about it within milliseconds. ## Dry run and diffing `?dryRun=All` (`kubectl apply --dry-run=server`) executes stages 1–6 and then discards instead of writing. Because it exercises defaulting and the real admission chain, it is a genuine preflight — unlike client-side dry run, which only parses YAML locally. `kubectl diff` uses exactly this to show what a change would actually do after defaults and webhooks.