What DNS records does a Kubernetes Service created with clusterIP: None produce, and how do StatefulSet pods obtain stable individual DNS names from it?

## What headless means A normal Service allocates a ClusterIP: a virtual address that never lives on an interface and is translated to a backing pod by the node's datapath. Setting `spec.clusterIP: None` disables all of that. There is no virtual address, no translation, and no proxying. The Service becomes purely a **grouping and naming** construct over a set of pods. Cluster DNS then changes behaviour. Instead of one A record for a virtual address, it answers the Service name with **one address record per ready backing pod**. A client resolving `cassandra.data.svc.cluster.local` receives the full membership list and connects directly to whichever members it wants. ## Why anyone wants this Three reasons dominate: 1. **Peer discovery.** Distributed databases and queues — Cassandra, etcd, Kafka, Elasticsearch, Zookeeper — need each node to know its peers. Resolving the headless name is a bootstrap mechanism that needs no external registry. 2. **Client-side load balancing.** Long-lived multiplexed connections, notably gRPC over HTTP/2, do not benefit from connection-level distribution: one connection carries everything and pins to one backend. Given all member addresses, a gRPC client can open a subchannel per address and balance per request. 3. **Addressing a specific member.** Sharded and primary/replica systems need to talk to *this* instance, not any instance — impossible through a single virtual address. ## Per-pod names with StatefulSets A StatefulSet declares `spec.serviceName: <headless-service>` — the "governing service". This gives each pod its own DNS name: ``` <pod-name>.<service-name>.<namespace>.svc.cluster.local ``` for example `kafka-1.kafka-headless.data.svc.cluster.local`. Because a StatefulSet assigns stable ordinal names, and re-creates a replacement under the same name, that DNS name is durable across reschedules even though the address behind it changes. This is the whole point: the identity is the name, and configuration files, seed lists and peer certificates can reference it. SRV records complete the picture: querying `_<port-name>._tcp.<service>.<namespace>.svc.cluster.local` returns one entry per member with its port and target name, which is how some clients enumerate a cluster in a single query. ## Readiness and the bootstrap deadlock By default only **ready** endpoints appear. That is usually right — you do not want clients steered at a member that cannot serve — but it creates a chicken-and-egg failure on first start: a quorum-based system's members are unready until they find each other, and they cannot find each other because unready members are not published. The result is a cluster that never forms. The fix is `publishNotReadyAddresses: true` on the headless Service, which publishes endpoints regardless of readiness. Use it deliberately, and only where the client tolerates dialling members that are still starting — which peer protocols generally do, since they retry. ## Caching pitfalls With a virtual address, a stale DNS answer is harmless: the address is stable and the datapath handles membership. With a headless Service, the DNS answer *is* the membership, so caching directly determines correctness. - CoreDNS serves these records with a short TTL, but a client that caches beyond it keeps dialling departed pods. - The classic offender is a JVM caching resolutions for the process lifetime under a restrictive security-manager-era default; long-running services must set a bounded cache TTL. - Connection pools that resolve once at startup have the same problem in a different layer: the pool holds addresses of pods that no longer exist and never re-resolves. So when a client keeps hitting a deleted replica after a rollout, look at cache TTL and pool re-resolution before suspecting the cluster. ## Other properties worth stating - **No ordering guarantee.** DNS returns the member set, not a ranking; do not treat the first answer as a primary. - **Selectorless headless Services** (no `selector`) let you manage EndpointSlices by hand, which is how an external system is given an in-cluster name that resolves to its real addresses. - **Not the same as ExternalName.** An ExternalName Service answers with a CNAME to an outside hostname and involves no pods at all. - **Policy still applies.** Discovering an address does not mean you may reach it; enforcement is separate. ## Answering well Say what the record set is, name the per-pod form and where it comes from, then show the two operational edges — readiness exclusion versus `publishNotReadyAddresses`, and client caching turning a membership answer stale. Those edges are what the interviewer is checking for.