How does an individual Pod in a Kubernetes StatefulSet get a stable DNS name, and what role does a headless Service play in that?

## Why a normal Service is not enough A standard ClusterIP Service allocates one virtual IP and load-balances connections across all Ready endpoints. That is ideal for stateless traffic and useless for a clustered database: if `kafka-0` needs to replicate to `kafka-1` specifically, resolving a name that lands on a random broker is wrong. Clustered systems need to address *members*, not the set. ## The headless Service Setting `spec.clusterIP: None` makes a Service headless. Concretely: - No virtual IP is allocated, and kube-proxy programs no forwarding rules for it. - Cluster DNS answers a query for the Service name with the IP addresses of the backing Pods themselves (multiple A/AAAA records), instead of a single virtual IP. - Because the endpoints are real Pod IPs, clients connect directly to Pods. On its own that already gives you discovery: one lookup, all the members. ## Per-Pod records The StatefulSet controller adds identity on top. It sets each Pod's `spec.hostname` to the Pod name (`kafka-1`) and `spec.subdomain` to the governing Service named in `spec.serviceName` (`kafka-hs`). Cluster DNS uses those fields to publish an individual record per Pod: ``` <pod-name>.<governing-service>.<namespace>.svc.cluster.local kafka-1.kafka-hs.data.svc.cluster.local ``` Inside the same namespace the short form `kafka-1.kafka-hs` resolves too, thanks to the Pod's DNS search path. The container's own `hostname` command also returns `kafka-1`, and its FQDN matches — which matters for systems like Kafka, Cassandra and ZooKeeper that advertise their own hostname to peers. The governing Service must actually exist, and the StatefulSet does not create it for you. A missing or misnamed `serviceName` is a classic bug: Pods start fine, but peer resolution silently fails and the cluster never forms. ## What is and is not stable The **name** is stable; the **IP is not**. When `kafka-1` is rescheduled onto a different node it gets a new Pod IP, and DNS is updated to match. This is the whole point: peers configured with `kafka-1.kafka-hs` keep working across node failures, rolling updates and cluster upgrades. Any application that caches a resolved IP indefinitely defeats it — JVM services in particular should have their DNS cache TTL set to something short rather than the historical forever. ## Readiness and bootstrap By default a headless Service publishes only *ready* endpoints, and per-Pod DNS records follow the same rule. That creates a bootstrap deadlock for systems where a node is not ready until it has joined a quorum, but cannot join a quorum until it can resolve its peers. The fix is `publishNotReadyAddresses: true` on the headless Service, which publishes records for Pods regardless of readiness so members can discover each other while still starting up. A common production layout uses **two Services**: a headless governing Service with `publishNotReadyAddresses: true` for peer-to-peer traffic and identity, and a separate normal ClusterIP Service for client traffic, which correctly load-balances over ready members only. Some systems add a third selector-labelled Service pointing only at the current primary. ## SRV records and discovery Headless Services also publish SRV records for their named ports, in the form `_<port-name>._<protocol>.<service>.<namespace>.svc.cluster.local`, resolving to the per-Pod names. Systems that discover peers dynamically use an SRV lookup rather than a hardcoded member list, so the set can grow without editing config. ## Practical checks When peer discovery breaks, verify in order: does the headless Service exist with exactly the name in `spec.serviceName`; does its selector actually match the Pod labels; are endpoints populated; does a DNS lookup from inside a Pod resolve both the Service name and an individual Pod name; and is readiness gating the records you expect. Nearly every StatefulSet DNS incident is one of those five.