A Deployment scales from 3 to 6 replicas and the new pods stay Pending with the scheduler message `node(s) didn't match pod topology spread constraints`. Explain what that constraint does and how you would decide between relaxing it and adding capacity.

## The goal: not all eggs in one basket Running six replicas means nothing for availability if all six land in one availability zone or on one node. Kubernetes offers two mechanisms to prevent that. ### podAntiAffinity — the older, blunt tool ```yaml affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchLabels: { app: api } topologyKey: kubernetes.io/hostname ``` This says: no two `app=api` pods may share a node. It is binary — one per domain, no more. On a five-node cluster, replica six is permanently `Pending` with `node(s) didn't satisfy existing pods anti-affinity rules`. It is also expensive to evaluate at scale, because the scheduler must inspect the pods on every candidate node. ### topologySpreadConstraints — the modern, tunable tool ```yaml topologySpreadConstraints: - maxSkew: 1 topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: DoNotSchedule labelSelector: matchLabels: { app: api } ``` Instead of "at most one", it bounds the **skew**: for the set of pods matching the selector, `skew = count(this domain) − count(least-populated eligible domain)`. Placing a pod is allowed only if the resulting skew stays within `maxSkew`. Worked example: three zones, `maxSkew: 1`, and existing counts `a=2, b=2, c=2`. Adding one pod makes that domain 3 against a minimum of 2 — skew 1, allowed. But if zone `c` has no schedulable capacity, placing pods only in `a` and `b` quickly gives `a=3, b=3, c=2`… then the next pod would make `a=4` against `c=2`, skew 2, forbidden. Every node fails the predicate and the pod stays `Pending` with `node(s) didn't match pod topology spread constraints`. The crucial insight: **the constraint is working as designed**. It is refusing to concentrate risk. `Pending` is the constraint reporting that you do not have the capacity to remain as available as you asked to be. ## The knobs that change the outcome - **`whenUnsatisfiable: ScheduleAnyway`** — demotes the rule from a filter to a scoring preference. Pods always schedule; balance is best-effort. This is the right choice when imbalance is worse-but-acceptable and downtime is not. - **`maxSkew`** — raising it to 2 or 3 tolerates more imbalance while still preventing total concentration. - **`minDomains`** — requires at least N eligible domains to exist; with fewer, the constraint fails. Useful to force multi-zone presence, but it can also be the hidden reason for `Pending` when a zone's nodes are all cordoned. - **`nodeAffinityPolicy` / `nodeTaintsPolicy`** (`Honor` / `Ignore`) — whether nodes excluded by the pod's own affinity or by taints still count as domains. `Honor` (the default for nodeAffinityPolicy) prevents a zone your pod could never use from being counted as "empty" and distorting the skew. - **`matchLabelKeys`** — commonly `[pod-template-hash]`, so a rolling update counts only pods from the same revision. Without it, old and new ReplicaSet pods are counted together and a rollout can deadlock: the old pods occupy the domains, so the new ones cannot satisfy the skew, so the old ones are never removed. ## Diagnosis 1. Read the full event — it names the constraint that failed and often the skew. 2. Count reality: `kubectl get pods -l app=api -o wide` grouped by node/zone tells you the current distribution instantly. 3. Check which domains are actually usable: cordoned nodes, taints, and zones with no free allocatable capacity all shrink the eligible set. 4. Look for a rollout in progress — a spread deadlock during an update is usually the `matchLabelKeys` problem or a `maxUnavailable: 0` strategy combined with a hard spread. ## The decision Ask what the constraint exists for. - If it encodes an availability requirement — "we must survive losing a zone" — then `Pending` is the honest signal that the cluster cannot host six replicas at that guarantee. **Add capacity in the under-populated domain** (or let the cluster autoscaler do it; modern autoscalers understand spread constraints and will scale the right zone's node group). Relaxing the constraint here silently converts an availability guarantee into an outage waiting for a zone failure. - If it is a best-effort balance for a stateless service where partial concentration is tolerable, use `ScheduleAnyway`. Running six imbalanced replicas beats running three. A good production pattern is layered: a hard constraint at `maxSkew: 1` across **zones**, and a soft (`ScheduleAnyway`) constraint across **hostnames** — guaranteed zone diversity, best-effort node diversity — and `matchLabelKeys: [pod-template-hash]` so rollouts do not deadlock. ## Interview framing Define skew precisely, show that `DoNotSchedule` makes it a filter, name the deciding question (is this an availability guarantee or a preference?), and mention the rollout deadlock and `matchLabelKeys`. That last detail is what separates someone who has operated this from someone who has read about it.