You are defining the pod priority tiers for a shared Kubernetes cluster running production services, internal tooling and opportunistic batch jobs. How do you choose the tiers and their values, and what failure modes do you design against?

## Start from the decision the scheme has to make A priority scheme answers one question: *when the cluster is short of capacity, whose work stops?* Everything else — the numbers, the names, the policies — is bookkeeping. So begin by writing down, in plain language, the ordering you would defend to the affected teams. If you cannot state it in one sentence per tier, the scheme is too complicated to hold during an incident. A typical defensible ladder: | Tier | Value | Preemption | Meaning | |---|---|---|---| | `platform-critical` | 1000000000 | PreemptLowerPriority | Ingress, service mesh control, logging pipeline — cluster-wide blast radius if down | | `production` | 1000000 | Never | Customer-facing services | | `internal` | 100000 | Never | Dashboards, dev tooling, CI runners | | `batch` | 1000 | Never | Scheduled ETL, reports | | `preemptible` | -10 | Never | Opportunistic work, explicitly disposable | Note the shape: only the top tier can evict, values are orders of magnitude apart, and the bottom tier is *negative* so it sits below anything unlabeled (priority 0). ## Design rules **Few tiers, wide gaps.** Gaps of 10× or more mean you can insert `production-tier-2` later without touching existing classes — important because edits do not re-price running pods, so renumbering produces a cluster with two generations of the same name at different values. **Low or zero globalDefault.** The default class is the one nobody chose, which usually means the pod that nobody thought about. Letting it inherit a high number gives every unreviewed workload eviction rights. Point `globalDefault` at `batch`, or leave it unset so unlabeled pods are 0. **Preemption is a separate grant from rank.** `preemptionPolicy: Never` keeps queue precedence while removing eviction power. Default every tier to `Never`, then justify each exception. In most clusters exactly one or two tiers need real preemption: the platform tier (so a DaemonSet-adjacent component can always start) and, sometimes, a bounded-latency job tier whose victims are all `preemptible`. **Make the victim tier cheap.** If a tier exists to be preempted, engineer it that way: `terminationGracePeriodSeconds: 10–30`, checkpointing or idempotent restart, a controller that requeues rather than failing the job, and no PodDisruptionBudget illusions. A 300-second grace period on a victim means the "urgent" preemptor waits five minutes. **Enforce eligibility mechanically.** PriorityClasses are cluster-scoped and referencing them is not gated by RBAC. Per-namespace `ResourceQuota` with `scopeSelector` on `PriorityClass` — either an allowlist with real pod counts, or a zero quota on forbidden classes — is the enforcement point. Convention plus code review is not enforcement. ## Failure modes to design against **Priority inflation.** Left ungoverned, every team labels itself `production` and the ladder collapses into one rung. Counter it with quota-based eligibility, an approval path with a named owner per class, and periodic audits (`kubectl get pods -A -o custom-columns=...,PRIO:.spec.priority --sort-by=.spec.priority`). **PDB mistaken for a guarantee.** Preemption ranks candidate nodes by fewest PodDisruptionBudget violations, but it will break a budget if that is the only way to schedule. Teams that treat a PDB as protection will be surprised. Document that PDBs bind drains (the Eviction API), not preemption. **Preemption thrash.** Victims belong to controllers that recreate them, which then get preempted again — CPU burned, nothing progresses, logs full of Preempted events. Detect it by alerting on preemption *rate* per namespace, and fix it either by capacity or by making the low tier back off. **Nomination is not a booking.** `status.nominatedNodeName` is a soft reservation; the freed room can be taken by someone else, and grace periods delay it. So "high priority" never means "starts immediately". Alert on pending duration against an SLO instead of assuming the tier delivers. **Priority substituting for capacity.** A cluster that preempts continuously is under-provisioned. Preemption is a redistribution mechanism, not a source of resources. The genuinely comparative question is preemption versus Cluster Autoscaler scale-up: preemption is instant-ish and free but destructive; scale-up is non-destructive but costs money and takes minutes. Bounded-latency work on a cost-sensitive cluster leans on preemption of a disposable tier; everything else should lean on capacity. **Priority is not QoS, and not isolation.** A `production` pod with no memory requests is still `BestEffort` and can be OOM-killed regardless of its priority. Tiering must be paired with a requests/limits policy, and hard isolation between tenants remains a matter of separate node pools, taints or separate clusters — priority only orders the queue and the executions. ## Rollout Ship the classes with `preemptionPolicy: Never` everywhere first and observe for a few weeks: who ends up in which tier, how long each tier waits, whether the ordering matches intuition. Then enable preemption on the single tier that needs it, with the victim tier already engineered to be disposable. That sequencing gives you the fairness benefit with zero eviction blast radius while the scheme is still wrong.