skip to content

A Kubernetes Service exists and its cluster DNS name resolves, but every connection to it fails and the Service has no backend addresses at all. How does a Service decide which pods back it, and how would you work out why the backend list is empty?

level: middleimportance: must knowfreq 62%

answer

  1. selector = equality AND, same namespace only
  2. DNS resolves the VIP even with zero backends
  3. only Ready addresses receive traffic
  4. named targetPort missing on pod -> no endpoint
  5. no selector = you write the backends yourself

basics

~20 s

A controller watches pods matching the Service's label selector in the same namespace and publishes the ready ones as the Service's backend addresses. An empty list means no pod matches the selector, matching pods are not Ready, or the pods are in another namespace.

solid answer

~50 s

A Service does not point at pods directly. A controller continuously lists pods in the **same namespace** whose labels match `spec.selector`, and publishes their IP/port pairs as the Service's backend set; only pods that are **Ready** (all readiness probes passing, not terminating) are put in the ready list that the dataplane load-balances over. That is why a Service can resolve in DNS while having zero usable backends. The checks, in order: 1. `kubectl get svc <name> -o yaml` and compare `spec.selector` with the pod labels from `kubectl get pods --show-labels`. A selector is an exact AND over all key/value pairs; one typo or an extra key matches nothing. 2. Namespace: selectors never cross namespaces. 3. `kubectl get pods` READY column. Pods that exist but fail readiness produce no ready endpoints. Check the probe path/port. 4. Port mismatch: a **named** `targetPort` that no container declares yields no endpoint for that port. A Service with **no** selector never gets endpoints automatically by design: you supply them yourself.

code

bash · 6 lines
bash
kubectl get svc api -o jsonpath='{.spec.selector}{"\n"}'
kubectl get pods --show-labels
kubectl get pods -l app=api,tier=backend
kubectl describe svc api
kubectl get endpointslices -l kubernetes.io/service-name=api -o yaml
kubectl describe pod api-7d9f -n prod | sed -n '/Readiness/,+3p'

go deeper

for a junior

Know that the label selector picks the pods and that mismatched labels or a wrong namespace produce an empty backend list.

for a middle

Add readiness gating, named-port resolution and the describe/endpointslice commands that show what was actually computed.

for a senior

Diagnose fast and in order, distinguish selector problems from readiness problems from app-bind problems, and know publishNotReadyAddresses and selector-less Services as deliberate tools.

for a principal

Push on label discipline as a platform concern: overly broad selectors silently merging workloads, probes that do not reflect dependency health, and templating that keeps Deployment and Service selectors in sync.

## The selector is the whole mechanism A Service manifest lists labels under `spec.selector`. A control-plane controller watches every pod and, for each Service, computes the set of pods in the **same namespace** whose labels are a superset of that selector. The selector is a plain equality match ANDed across all pairs: `{app: api, tier: backend}` matches a pod labelled `app=api, tier=backend, version=v3`, but not one labelled only `app=api`. Service selectors do not support set-based expressions (`In`, `NotIn`, `Exists`) the way Deployment and NetworkPolicy selectors do. The resulting pod IP/port pairs are written into backend objects that the per-node dataplane consumes; when they are empty, the node has no rule to send traffic anywhere and connections to the virtual IP are refused or time out. Crucially the **DNS record still exists** for a Service with a virtual IP, because DNS answers with the VIP, not with the pod set. So "the name resolves" tells you nothing about whether backends exist - a distinction that trips people up constantly. ## Ready versus not ready The backend record for each address carries a readiness condition. Only addresses marked ready receive normal traffic. An address becomes not-ready when a readiness probe fails, when the pod enters terminating state, or before startup completes. There is also a **terminating** flag, so implementations can keep draining in-flight connections to a pod that is shutting down. This is the second most common cause of an empty backend set: the pods are running, the labels match, but readiness never turns true because the probe points at the wrong port or path, or the app takes longer to start than the probe's failure budget allows. A related knob is `publishNotReadyAddresses: true`, which forces addresses to be published regardless of readiness. It exists for cases such as headless Services fronting clustered software that must discover peers *before* any of them is ready. ## Walking the diagnosis Start from the Service and move toward the pod. 1. **Selector vs labels.** `kubectl get svc api -o jsonpath='{.spec.selector}'` and `kubectl get pods --show-labels`. Compare character by character; `app: api` vs `app: API` fails silently. Deployments make this easy to get wrong because the Service selector and the Deployment's `spec.selector.matchLabels` are separate objects that nothing forces to agree. 2. **Namespace.** `kubectl get pods -n <svc-namespace> -l <selector>`. A Service in `prod` will never select a pod in `staging`. To reach across namespaces you address the other Service's DNS name; you do not widen a selector. 3. **Readiness.** `kubectl get pods -l <selector>` and read the READY column, then `kubectl describe pod` for probe failures. If probes fail, fix the probe or the app, not the Service. 4. **Ports.** If `targetPort` is a name, each pod must declare a `containerPort` with that name or it contributes no endpoint. If it is a number, the app must actually listen on it and on `0.0.0.0`, not `127.0.0.1`. 5. **Confirm the result.** `kubectl get endpointslices -l kubernetes.io/service-name=api -o yaml` shows exactly what the controller computed: addresses, ports and per-address ready conditions. `kubectl describe svc api` prints a condensed view of the same thing. ## Services with no selector Omitting `spec.selector` is legal and meaningful: it says "I will manage the backends myself". You then create the backend object by hand, pointing at arbitrary IPs - typically an external database or a service in another cluster. The Service keeps a virtual IP, in-cluster load balancing and port remapping, which is what distinguishes it from an ExternalName Service (pure DNS alias, no proxying, no port remapping). If you see an empty backend list on a selector-less Service, nobody created the hand-written backends, or a label required to associate them with the Service is missing. ## Why the mapping is dynamic, and what that costs Because membership is recomputed from labels on every pod change, rollouts are automatic: a new pod that gets the right labels and passes readiness joins the pool with no Service edit. The cost is that the pool is only as correct as your labels and probes. Two mistakes worth naming: giving a Service a selector so broad that it also matches pods from a different Deployment (mixing versions or even applications behind one name), and having a readiness probe that always returns 200 regardless of dependency health, which keeps a broken pod in rotation.

  • The Service name resolves in DNS even though it has no backends. Why does DNS not reflect the missing backends?
    For a normal Service, cluster DNS returns the Service's virtual IP, which is allocated at creation time and is independent of the backend set. Resolution therefore succeeds whether or not any pod is selected; the failure only appears at connect time. A headless Service behaves differently: it publishes the ready pod IPs as A/AAAA records, so an empty backend set does show up as NXDOMAIN or an empty answer.
  • When would you deliberately set publishNotReadyAddresses: true?
    For clustered software that must discover its peers before any member can become ready, such as a quorum-based database bootstrapping through a headless Service. Publishing not-ready addresses lets members resolve each other during startup. It is a poor choice for normal request-serving Services, because clients would then be sent to pods that have not finished starting.

saying these in an interview costs you the question

  • Assuming a resolvable DNS name proves the Service has working backends.
  • Thinking a Service selector can match pods in another namespace.
  • Expecting set-based selector expressions (In/NotIn/Exists) to work in a Service selector.
  • Forgetting that failing readiness probes remove pods from the ready pool while the pods still look Running.
  • Treating a selector-less Service as broken instead of as a manual-endpoints Service.

context

resolving links…