How would you use Kubernetes podAntiAffinity to keep the replicas of a Deployment off the same machine, and what exactly does the topologyKey field control?

## What inter-Pod affinity is Node affinity answers "what kind of machine do I need?". Inter-Pod affinity and anti-affinity answer "who else is already there?". `podAffinity` attracts a Pod towards nodes where matching Pods run; `podAntiAffinity` repels it away. Both live under `spec.affinity` and both have the same required/preferred split as node affinity. ## The three fields of a term 1. **labelSelector** — which existing Pods count. Standard `matchLabels`/`matchExpressions`. For spreading a Deployment's replicas you select the Deployment's own Pod template labels. 2. **namespaces** / **namespaceSelector** — which namespaces those Pods are looked for in. Omitted, it means the scheduling Pod's own namespace. This trips people up: a rule intended to be cluster-wide silently applies to one namespace. 3. **topologyKey** — mandatory and non-empty. It names a node label, and nodes carrying the same value for that label form one *topology domain*. ## How topologyKey actually evaluates For a candidate node N, the scheduler reads N's value for the topologyKey label — say `topology.kubernetes.io/zone=eu-west-1a`. It then looks at every Pod matching the labelSelector across the cluster, finds their nodes, and reads those nodes' values for the same label. For hard anti-affinity, if any matching Pod sits on a node with value `eu-west-1a`, node N is filtered out. So the unit of exclusion is the *domain*, not the node. Common topologyKeys: - `kubernetes.io/hostname` — one Pod per node. - `topology.kubernetes.io/zone` — one Pod per availability zone. - `topology.kubernetes.io/region` — one Pod per region. - Any custom label you apply consistently, e.g. `rack`, `failure-domain`. A node missing the topologyKey label is not part of any domain and is skipped for anti-affinity purposes — an unlabelled node pool is a silent hole in your spreading guarantee. ## Hard versus soft `requiredDuringSchedulingIgnoredDuringExecution` is a filter: the domain is either allowed or not. Its value is that the guarantee is real. Its cost is that the number of schedulable replicas is capped by the number of domains. Three replicas with hard hostname anti-affinity on a two-node cluster means one Pod stays `Pending` indefinitely, and a rolling update can deadlock because the new Pod cannot be placed until the old one on that node goes away — with the default `maxSurge` strategy that is a real hang. `preferredDuringSchedulingIgnoredDuringExecution` wraps each term in `{weight: 1-100, podAffinityTerm: {...}}`. Matching domains lose score, so the scheduler spreads if it can and packs if it must. For most stateless services this is the right default; for quorum-based systems (etcd, ZooKeeper, a database with synchronous replicas) where two replicas on one machine defeats the point, hard is justified. A frequent production pattern is both at once: hard anti-affinity at zone level for correctness, soft at hostname level for extra resilience. ## podAffinity, the mirror image The same structure attracts instead of repels: put a cache Pod in the same zone as the service that reads it by selecting `app=api` with `topologyKey: topology.kubernetes.io/zone` to cut cross-zone traffic charges and latency. Hostname-level podAffinity is rarely wise — it packs Pods onto one machine and turns that machine into a correlated failure domain. ## Practical rules - Anti-affinity and affinity are evaluated only at scheduling. Adding nodes later does not rebalance existing Pods; you must delete Pods or run a descheduler. - The `LimitPodHardAntiAffinityTopology` admission plugin, where enabled, forces `topologyKey: kubernetes.io/hostname` for required anti-affinity, precisely because arbitrary topologyKeys on hard rules are expensive. - Kubernetes offers a purpose-built alternative for even spreading; anti-affinity is the on/off tool, not the balancing tool. ## Diagnosis A blocked Pod's `FailedScheduling` event reads like "0/6 nodes are available: 3 node(s) didn't match pod anti-affinity rules, 3 node(s) had untolerated taint". That message tells you how many nodes each predicate eliminated, which is usually enough to see whether the anti-affinity or something else is the binding constraint.