A Kubernetes Deployment declares a topology spread constraint with `maxSkew: 1` across availability zones, yet after a rollout its pods sit 5/2/1 across three zones. Give the likely causes and how you would investigate.

## The governing fact: spreading is a placement-time decision `topologySpreadConstraints` is evaluated by the scheduler when it binds a pod to a node. It is never re-evaluated. There is no controller reconciling the distribution back towards balance. Every drift explanation follows from that: whatever the cluster looked like at bind time is what the constraint was judged against, and any later change — a zone recovering, nodes being added, other pods disappearing — leaves the existing placement untouched. ## Cause 1: the constraint is soft `whenUnsatisfiable: ScheduleAnyway` only contributes score. Resource-fit, image locality, affinity preferences and taint scoring are all summed alongside it, and on a cluster where one zone has much larger or emptier nodes the spread score can lose repeatedly. The result is exactly the kind of lopsided distribution described. Read the manifest as deployed (`kubectl get deploy -o yaml`) rather than trusting the chart's intent, and check whether a Helm value or an admission mutator changed it. ## Cause 2: domains were not equally able to accept pods Spread constraints do not create capacity. If zone C had no allocatable room, or its nodes carried a taint the pod does not tolerate, or its nodes failed the pod's own nodeSelector, the scheduler either skipped it (soft) or left pods Pending (hard). With `nodeTaintsPolicy: Ignore` — the default — nodes the pod cannot tolerate are still counted as part of the domain, which makes a domain look emptier than it usably is and can distort decisions. Setting `nodeTaintsPolicy: Honor` and `nodeAffinityPolicy: Honor` makes the computation reflect nodes the pod could actually use. ## Cause 3: nodes missing the topology label A node without `topology.kubernetes.io/zone` belongs to no domain. It is excluded from the skew computation, and pods that land on it are invisible to the counts. Mixed clusters — on-prem nodes added to a cloud cluster, or a node group whose bootstrap script omits the labels — produce distributions that look impossible until you check. `kubectl get nodes -L topology.kubernetes.io/zone` is a one-line diagnosis. ## Cause 4: the labelSelector counts the wrong population The selector is evaluated against all pods in the namespace, not just this ReplicaSet's. Two consequences: - **Other workloads sharing a label** are counted, so your balance is computed over a bigger, differently-distributed population. - **During a rolling update**, old and new ReplicaSet pods both match `app: web`. The scheduler counts them together, so the new pods are placed to balance the *combined* population; as the old pods are then deleted, the new set alone can be badly skewed. `matchLabelKeys: [pod-template-hash]` fixes this by having the scheduler add the incoming pod's `pod-template-hash` value to the selector, scoping the count to the current revision. ## Cause 5: scale-down and eviction chose badly The scheduler places pods; it does not choose which pods die. Scale-down picks victims by the ReplicaSet controller's own ordering, node drains remove whole nodes' worth of pods, and preemption evicts by priority. None of them consult topology spread. So a Deployment scaled 9→3 can end with all three survivors in one zone even though the original nine were perfectly balanced. ## Investigation procedure 1. **Measure.** Join pods to their nodes' zone labels and count — do not eyeball `kubectl get pods -o wide`. 2. **Read the effective constraint.** `kubectl get deploy web -o jsonpath='{.spec.template.spec.topologySpreadConstraints}'`. Confirm `whenUnsatisfiable`, `maxSkew`, `topologyKey`, and the selector. 3. **Check node labels.** `kubectl get nodes -L topology.kubernetes.io/zone` — any blanks are excluded domains. 4. **Check the selector's real population.** `kubectl get pods -l app=web -o wide | wc -l` versus the ReplicaSet's replica count; a mismatch means you are balancing more than you thought. 5. **Check Pending pods' events.** `didn't match pod topology spread constraints` proves a hard constraint is biting; the absence of Pending pods with a lopsided result points at a soft constraint or historic placement. 6. **Check for taints/capacity in the light zone.** `kubectl describe node` in the underweight zone. ## Fixing Correct the constraint first — tighten to `DoNotSchedule` if the workload genuinely needs it, add `matchLabelKeys`, fix node labels, set `nodeTaintsPolicy: Honor`. Then force re-placement: `kubectl rollout restart deployment/web` recreates pods so the corrected constraint is applied. For ongoing drift, run a descheduler with the `RemovePodsViolatingTopologySpreadConstraint` strategy so pods violating the constraint are evicted periodically and rescheduled — with a PodDisruptionBudget in place so the eviction is safe. Add an alert on observed per-zone counts; balance is not something you can assume once and forget, because nothing in the core control plane maintains it.