A Kubernetes Service of type ClusterIP gets an IP address that no network interface owns and that nothing answers ARP for. Explain what actually happens to a packet sent to that IP, and which component makes it work.

## The ClusterIP is deliberate fiction When you create a Service, the API server allocates an address from the **Service CIDR**, a range configured on the control plane (commonly something like `10.96.0.0/12`). That allocation is a record in etcd and nothing more. No node configures the address on an interface, no ARP or NDP responder claims it, and pinging it usually fails even when the Service works perfectly. The address is a *label the kernel matches on*, not a destination that exists. ## kube-proxy's job kube-proxy runs on every node. It watches the API server for two kinds of object: - **Services**, which give it the virtual IP, ports, protocol and session-affinity settings. - **EndpointSlices**, which give it the current set of backend pod IPs and their readiness. From those it computes the rules the node's kernel needs and writes them into the chosen dataplane. It then reconciles: every time endpoints change, it recomputes and re-syncs. Crucially, kube-proxy does not forward traffic. Once the rules are installed the kernel does all the work at line rate, which is why a crashed kube-proxy does not immediately break existing Services - it just stops them from being updated, so the rules go stale. ## What happens to the packet Say a pod at `10.244.1.7` connects to a Service at `10.96.0.10:80` whose backends are `10.244.2.3:8080` and `10.244.3.9:8080`. 1. The pod's socket sends a SYN to `10.96.0.10:80`. Routing sends it toward the node's stack. 2. In the NAT path (iptables `nat` OUTPUT/PREROUTING, or the IPVS/nftables equivalent) a rule matches destination `10.96.0.10` and TCP port 80. 3. One backend is selected. In iptables mode this is done with a chain of `statistic mode random probability` jumps that produce an approximately even split; in IPVS mode a real scheduler (round robin by default) picks it. 4. **DNAT** rewrites destination address and port to `10.244.2.3:8080`. The source is untouched for in-cluster traffic, so the backend sees the real client pod IP. 5. The kernel's **conntrack** table stores the tuple with its translation. Every subsequent packet of that flow follows the same entry - hence per-connection, not per-packet, load balancing - and reply packets get reverse-NATed so the client sees replies from `10.96.0.10:80`, which is what the client's socket expects. ## Consequences that show up in interviews - **Load balancing is L4 and per-connection.** A client that opens one long-lived HTTP/2 or gRPC connection pins to one backend forever. Fixing that needs client-side balancing, a headless Service, or a service mesh / L7 proxy - not kube-proxy. - **`ping` a ClusterIP fails** because rules typically match only the declared protocol and ports. Testing with `curl <clusterip>:<port>` is the correct check. - **No ready endpoints** means the rule set has nothing to DNAT to; iptables mode installs a REJECT so you get connection refused fast rather than a hang. - **Session affinity** (`service.spec.sessionAffinity: ClientIP`) is the only stickiness kube-proxy offers, implemented with a client-IP timeout in the same kernel machinery. - **Headless Services** (`clusterIP: None`) opt out of all of this: no virtual IP, no DNAT, DNS just returns pod IPs directly. ## Where this fits The pod-to-pod substrate (routing pod IPs between nodes) is the CNI plugin's job. kube-proxy sits on top: it turns a stable Service name and virtual IP into one of those pod IPs. Some modern CNIs (Cilium in kube-proxy-replacement mode, for example) implement the same Service semantics in eBPF and let you delete kube-proxy entirely - the abstraction stays identical from the workload's point of view.