In a Docker Swarm cluster, how does a client request reach a service replica when it hits a published port on a node that is running no replica of that service, and how do containers on different hosts reach each other?

## Overlay networks An overlay network gives containers on different hosts one flat layer-2 segment. Each host has a hidden network namespace holding a bridge and a VXLAN interface; a packet from container A to container B on another node is bridged, encapsulated in a VXLAN header, sent host-to-host over **UDP 4789**, decapsulated on the far side and delivered. Membership and endpoint information propagate over the **gossip control plane on 7946 (TCP and UDP)**, while cluster management uses **2377/TCP** to the managers. Any firewall exercise on a swarm comes down to those three. By default the data path is unencrypted; `docker network create -d overlay --opt encrypted appnet` adds IPsec between nodes at a throughput cost. The control plane is already mutually authenticated with TLS. ## Discovery and internal load balancing Every service attached to an overlay network gets an entry in the swarm's embedded DNS. In the default **VIP endpoint mode**, the name resolves to a single virtual IP that belongs to no container. Traffic to the VIP is intercepted by **IPVS** in the kernel on the sending node and distributed across the task IPs, so failover and scaling need no client awareness and no DNS cache expiry. In **dnsrr endpoint mode** (`--endpoint-mode dnsrr`) the DNS answer contains the task addresses instead and the client picks. It suits clients that do their own pooling or need per-connection control, and it is required with host-mode publishing. The tradeoff is that clients cache DNS answers and can keep hitting departed tasks. `tasks.<service>` always resolves to the individual task IPs regardless of endpoint mode, which is handy for peer discovery in clustered applications. ## The routing mesh When a service publishes a port in the default `ingress` mode, swarm attaches it to a special overlay network called `ingress` and installs rules on *every* node so the published port is open there. A request to any node, replica or not, is destination-NAT'd into the ingress network and load balanced by IPVS to a task somewhere in the cluster. Operationally this is excellent: an external load balancer or DNS record can point at every node without tracking placement. The well-known drawback is that the client's source IP is lost — traffic is source-NAT'd on the way in, so access logs show an internal address and any IP-based logic breaks. Remedies: terminate at a proxy that adds `X-Forwarded-For` or the PROXY protocol, or publish with `mode=host`, which binds the port directly on nodes running a task, skips the mesh and preserves the client address. Host mode means only those nodes answer, so you need an external load balancer that knows where tasks are, and it caps you at one task per node per port. ## Diagnosing - `docker service inspect --format '{{json .Endpoint}}' <svc>` shows the VIP and publish mode. - `docker network inspect <overlay>` on a node running a task lists peers and container endpoints. - From a container on the network, `nslookup <service>` (VIP) and `nslookup tasks.<service>` (task IPs) separate DNS problems from load-balancing problems. - If cross-host traffic fails but same-host works, suspect UDP 4789 blocked or an MTU problem: VXLAN adds 50 bytes of header, so a path that cannot carry the resulting frames produces the classic symptom of small requests succeeding and large ones hanging. ## Summary sentence for an interview Overlay is VXLAN over UDP 4789 with gossip on 7946; discovery is DNS to a virtual IP with IPVS behind it; published ports use the ingress network so every node accepts and forwards — at the price of the client source IP unless you publish in host mode.