How does Docker's `overlay` network driver let containers on different hosts communicate as if on one network, and what does it require from the underlying infrastructure?

## The problem Bridge networks are host-local: container IPs are meaningful only on the host that allocated them, and two hosts will happily hand out the same `172.18.0.2`. To let containers on different machines address each other directly you either teach the physical network about container subnets (routing, which needs infrastructure cooperation) or you **tunnel** - which is what the overlay driver does. ## VXLAN in one paragraph VXLAN (Virtual eXtensible LAN) encapsulates a complete layer-2 Ethernet frame inside a UDP datagram. Each virtual network gets a 24-bit **VNI**, carried in the VXLAN header, so many isolated virtual networks share one physical fabric. The endpoints doing the wrapping and unwrapping are **VTEPs** - here, the Docker hosts. On the wire your infrastructure sees only UDP between host addresses on port 4789; the container-to-container frame is the payload. Inside each participating host, Docker creates a dedicated network namespace per overlay network holding a bridge and a `vxlan` interface; each container's veth lands in that namespace's bridge. Traffic to a local peer stays on the bridge; traffic to a remote peer goes out through the VXLAN interface. ## The control plane Encapsulation alone is not enough - a host must know that container IP `10.0.1.7` currently lives on host B, and the corresponding MAC. Docker fills the forwarding and ARP tables from cluster state rather than by flooding. In current Docker that state comes from **swarm mode**: managers hold the network and endpoint definitions in the Raft-backed store, and nodes exchange endpoint updates over a gossip protocol. (Historically, standalone overlay networks used an external key-value store such as Consul or etcd; that path is legacy.) This is why the ports matter and why a partially blocked firewall produces confusing symptoms: block **2377** and nodes cannot join or receive network definitions; block **7946** and membership and endpoint propagation degrade, so some pairs of containers resolve and connect while others do not; block **4789** and the control plane looks perfectly healthy while no data flows at all. ## MTU Encapsulation costs roughly 50 bytes of header. If the underlay MTU is 1500, the overlay must use about 1450. Docker sets a reduced MTU on overlay interfaces, but the failure mode when something is inconsistent - a tunnel or VPN in the underlay with its own reduced MTU, or mixed manual configuration - is distinctive: the TCP handshake and small requests succeed, and anything that fills a full-size packet (a large POST, a TLS certificate chain, a big query result) hangs or times out. If ICMP "fragmentation needed" is filtered, path MTU discovery cannot repair it. Suspect MTU whenever "small works, large hangs". ## Encryption and isolation Overlay control traffic is encrypted by swarm. **Data** traffic is not, unless the network is created with `--opt encrypted`, which sets up IPsec between the hosts. That is a real CPU cost at high throughput, so it is a deliberate choice - typically yes when hosts communicate over untrusted or shared networks, often no inside a trusted private network where the underlay is already protected. Each overlay network is a separate VNI, so networks are isolated from one another in the same way separate bridges are. Overlay networks also get the usual embedded DNS behaviour, so containers address each other by name across hosts. ## Attachability and scope By default overlay networks created in swarm mode are usable only by swarm **services**. Creating one with `--attachable` lets plain `docker run` containers join, which is handy for debugging or mixed deployments - at the cost of a wider surface, since any container on any node can join. ## When to use it Overlay is the answer when you must keep multi-host container networking inside Docker's own tooling. If you are already moving to a full orchestrator, its cluster networking layer solves the same problem with its own plugins, and the concepts transfer directly: encapsulation, a control plane distributing endpoints, MTU headroom, and optional encryption.