A Kubernetes StatefulSet's replicas report Ready only after forming a quorum, and they never do. Why, and what does publishNotReadyAddresses on its headless Service change?

## The deadlock Some clustered applications need their members to **find each other before any of them is useful**. Take a recommendation-model inference server, `rec-infer`, run as a 3-replica **StatefulSet**. Replicas elect a shard coordinator by majority vote, so at least 2 of the 3 must be talking. Each replica's **readiness probe** passes only after that vote succeeds and its 4.2 GB model shard is loaded (about 97 seconds on a single-node development cluster on a laptop). The replicas find their peers through the per-pod names of the StatefulSet's **headless Service**, such as `rec-infer-1.rec-infer-peers.ranking.svc.cluster.local`. Now follow the chain: 1. Cluster DNS publishes a per-pod record only for endpoints whose **`ready`** condition is true. 2. The EndpointSlice controller sets `ready` from the pod's readiness. 3. Readiness waits for a quorum. 4. A quorum needs peers to resolve each other's names. Nobody becomes ready, so nobody is published, so nobody becomes ready. The pods run, their logs show lookups that return no such name, and the set never forms. ## What `publishNotReadyAddresses` changes `spec.publishNotReadyAddresses` is a boolean on the **Service**. It tells anything that consumes that Service's endpoints to disregard readiness. Concretely, the EndpointSlice controller writes every endpoint of that Service with **`ready: true`**, whatever the pod's readiness probe says. The `serving` condition still reports the real probe result, so the truth is not lost. Because DNS follows the `ready` condition, each replica's per-pod name resolves as soon as the pod has an IP. The peers find each other, the vote succeeds, and readiness passes. Limits of the fix: - **An IP is still required.** A pod whose sandbox is not set up yet has no IP and gets no endpoint at all. - **Finished pods are dropped.** Pods in a terminal phase (Succeeded or Failed) are not listed. - **Terminating pods stay "ready".** With the flag set, a pod that is shutting down is still written with `ready: true`, so peers keep resolving a departing member. The application's own membership protocol has to cope with that. - **Negative answers get cached.** A replica that looked a peer up before the peer had an IP may have cached the failure. Lookups should retry with backoff. ## Why you want two Services The flag applies to **every consumer** of that Service. If clients used the same Service, they would be sent to replicas that are still loading. So the usual layout is: | Service | `clusterIP` | `publishNotReadyAddresses` | Used by | |---|---|---|---| | `rec-infer-peers` | `None` | `true` | Replicas finding each other; named in the StatefulSet's `spec.serviceName` | | `rec-infer` | assigned | `false` (default) | Ranking clients, which should only reach ready replicas | ```yaml apiVersion: v1 kind: Service metadata: name: rec-infer-peers namespace: ranking spec: clusterIP: None publishNotReadyAddresses: true selector: app: rec-infer ports: - name: peer port: 7351 --- apiVersion: v1 kind: Service metadata: name: rec-infer namespace: ranking spec: selector: app: rec-infer ports: - name: grpc port: 8471 targetPort: grpc ``` ## Checking it on a live cluster - List the peer Service's EndpointSlices with the `kubernetes.io/service-name` label and compare each endpoint's `ready` and `serving` conditions. With the flag working, an unready replica shows `ready: true` and `serving: false`. - Resolve a peer's per-pod name from inside another replica. - If a replica's endpoint is missing entirely, check that the pod has an IP and that the Service selector matches its labels. ## One more trap outside DNS Under the default StatefulSet `podManagementPolicy` of `OrderedReady`, the controller does not create `rec-infer-1` until `rec-infer-0` is Ready. If readiness needs a quorum, that alone is a deadlock that no DNS setting can fix. Quorum systems whose readiness depends on peers therefore usually run with `Parallel` pod management. How the StatefulSet orders pods belongs to StatefulSet design; the point here is to recognize both deadlocks, because they look alike from the outside. ## Deciding what readiness should mean The flag is a workaround for a design choice: readiness that depends on peers. Before reaching for it, ask what the readiness probe is for. If it means "this replica can answer ranking clients", a quorum check belongs in it, and the two-Service layout above is the right answer. If a replica can serve some traffic alone, a narrower probe removes the deadlock without any Service change. Either way, write down which Service each consumer uses, because the flag's effect is invisible from the pod side.