You are asked to move existing production namespaces to the Kubernetes restricted Pod Security Standard without breaking running workloads. How do you sequence that migration, and how do you diagnose workloads that stop being admitted?

## What makes this a migration rather than a flag flip The restricted Pod Security Standard differs from baseline in kind: baseline forbids dangerous fields, restricted *requires* hardening fields. That means almost every pre-existing workload fails restricted on day one, not because it does anything dangerous but because it never wrote `allowPrivilegeEscalation: false`. So the migration is a manifest-editing campaign with a policy switch at the end, and the risk is entirely in workloads you did not think to test. ## Phase 1 — observe without breaking Apply `pod-security.kubernetes.io/warn=restricted` and `pod-security.kubernetes.io/audit=restricted` to every namespace. Neither blocks anything. - **warn** returns a warning on the API response and, importantly, evaluates workload resources (Deployment, StatefulSet, DaemonSet, Job, CronJob) — so a developer running `kubectl apply` or a CI pipeline sees it at the moment they change something. - **audit** writes an annotation on the API server audit log entry, giving you a queryable fleet-wide inventory even for workloads nobody has touched in months. Also set the cluster-wide PSA admission-configuration default so newly created namespaces inherit the target level instead of being unrestricted. ## Phase 2 — measure honestly The trap is sampling only what redeploys. Run the observation window long enough to cover: - CronJobs with weekly or monthly schedules - Jobs triggered by incidents or batch runs - DaemonSets that only reschedule on node turnover - Anything installed by a Helm chart you do not own — third-party charts are the most common source of unfixable violations, and you need to know early whether the chart exposes a `securityContext` value or must be forked/replaced An alternative to waiting is to enumerate statically: pull every pod template from the API and evaluate it offline against the restricted rules. ## Phase 3 — fix the manifests The standard patch, applied at pod level where possible and repeated for every container including init and sidecars: - `runAsNonRoot: true` plus a numeric `runAsUser` (a named `USER` in the image is not resolvable by the kubelet's check) - `allowPrivilegeEscalation: false` - `capabilities: { drop: ["ALL"] }`, adding back only `NET_BIND_SERVICE` when a low port is unavoidable - `seccompProfile: { type: RuntimeDefault }` - volume types restricted to the safe list; replace `hostPath` with a PVC, a projected volume or a node-level agent that legitimately lives in a privileged namespace Expect collateral work: images that write into the image layer, apps that bind port 80, JVMs whose temp dir is unset, files in the image owned by root and unreadable by the new UID. `fsGroup` fixes volume ownership; a `chown` in the Dockerfile fixes image paths. ## Phase 4 — verify before enforcing `kubectl label --dry-run=server --overwrite ns <ns> pod-security.kubernetes.io/enforce=restricted` runs the policy against the namespace's existing pods and reports the violators without persisting the label. This is the go/no-go check. Do it per namespace immediately before flipping. Pin the version: `pod-security.kubernetes.io/enforce-version: v1.31`. Without a pin the level tracks `latest`, so a control-plane upgrade can introduce a new requirement that starts rejecting pods during an unrelated maintenance window. ## Phase 5 — enforce progressively Flip namespace by namespace, least critical first, with a rollback (remove the enforce label) rehearsed. Keep `warn` and `audit` set at the same level afterwards so drift is still visible. Infrastructure namespaces that genuinely need host access get an explicit `enforce: privileged` with an owner and a justification recorded — an explicit exception you can review beats an unlabelled namespace you forget. ## Diagnosing failures Split failures into two layers: **Admission-time (API server rejects the pod).** Because enforce acts on pods, the object you applied usually succeeds and the failure hides one level down. Check `kubectl describe replicaset <rs>`, `kubectl describe job`, or `kubectl get events --field-selector reason=FailedCreate`. The message enumerates the violated policy fields verbatim — `allowPrivilegeEscalation != false`, `unrestricted capabilities`, `runAsNonRoot != true`, `seccompProfile` — so it is a checklist, not a mystery. **Runtime (pod admitted, container will not run).** - `CreateContainerConfigError` with a root message → the kubelet's `runAsNonRoot` check; the image's user is UID 0 or an unresolvable name. - `Permission denied` on a write → `readOnlyRootFilesystem` and a missing writable mount. - `Operation not permitted` on a syscall → a dropped capability or the seccomp profile; identify it by running once with a permissive profile and tracing. - CrashLoopBackOff right after the change with a bind error → the app is trying to bind a port below 1024 without `NET_BIND_SERVICE`. ## Signals of success Zero audit annotations at the target level, every namespace explicitly labelled (no implicit defaults), a short reviewed list of privileged exceptions, and enforcement pinned to a version you upgrade deliberately.