Containers on a user-defined Docker bridge stall on large responses while small requests succeed — how do you diagnose it?

## Recognising the signature The symptom set is very specific and worth memorising, because it is easy to misdiagnose as an application bug: * TCP connections **establish** normally — the handshake packets are tiny. * Small requests and small responses work — `curl` against a health endpoint is fine. * Anything that pushes a full-size segment hangs: a TLS handshake carrying a large certificate chain, a multi-megabyte download, a big POST body. * The hang ends in a timeout, not a connection refused, and the application logs show nothing but a stalled read. That is an MTU problem until proven otherwise. ## Why the MTU matters here MTU is the largest payload a link will carry in one frame. A Docker bridge is an ordinary Linux bridge, and containers attached to it get an `eth0` whose MTU comes from the network's configuration — 1500 unless told otherwise. If the host then sends that packet out over an interface with a smaller MTU (a WireGuard or IPsec tunnel, a cloud overlay, a PPPoE link), the packet is too big for the next hop. In the normal case the router replies with an ICMP "fragmentation needed" message, the sender lowers its path MTU and everything recovers. In practice that ICMP is very often dropped by a firewall or a cloud security group, and you get a **path-MTU black hole**: the sender keeps retransmitting a segment nobody will forward. Because the container's own stack knows only about its 1500-byte `eth0`, the whole thing looks, from inside the container, like the peer went silent. ## Worked example A 47-node CI fleet runs the integration suite of a Ruby email-digest builder. Each job creates a per-build network, starts the worker and a Postgres container on it, and the worker pulls fixtures from an internal artefact host reached over a WireGuard tunnel whose interface MTU is 1412. Small API calls in the suite pass; the step that downloads a 38 MB fixture archive hangs and the job is killed by its 15-minute timeout. It reproduces on 11 of the 47 nodes — exactly the ones whose egress goes through the tunnel — which is what makes people blame "flaky nodes" instead of the network settings. ## Diagnosis, in order 1. **Establish that it is not name resolution.** The connection opens, so the peer's address was resolved and reachable. That already rules out a whole class of causes. 2. **Read the MTUs on both sides of the hop.** Inside a container on the network: `docker run --rm --network digest-net alpine ip -o link show eth0`. On the host: `ip -o link show` and find the interface the route to the peer uses (`ip route get <peer-ip>`). A container at 1500 leaving through a 1412 interface is the finding. 3. **Check what the network was created with.** `docker network inspect -f '{{index .Options "com.docker.network.driver.mtu"}}' digest-net` — an empty result means the option was never set. 4. **Confirm with a sized probe.** Send progressively larger payloads and find the cliff; the boundary will sit at the smaller MTU minus headers. 5. **Watch the retransmits.** A packet capture on the host shows the same large segment being retransmitted with no acknowledgement and no ICMP reply coming back. ## The fix Set the MTU on the network, at creation: ``` docker network create --opt com.docker.network.driver.mtu=1412 digest-net ``` A Docker network's options cannot be edited after creation, so applying this to an existing network means creating a replacement and re-attaching (or, for a stack, recreating it). For containers on the **default** bridge the equivalent is the daemon-wide `"mtu"` key in `/etc/docker/daemon.json` followed by a dockerd restart — which is exactly the granularity argument for not putting long-lived workloads on docker0 in the first place. Whatever the engine's default happens to be on a given version, do not assume a bridge you create has inherited the host's lower MTU. Verify it with `ip link` inside a container on that network; on a fleet, make that verification part of the node's build so a new image or kernel cannot silently change it. ## Prevention on a fleet * Bake the MTU option into whatever creates networks, so every node agrees. * Add a start-up check that compares the container MTU against the egress interface and fails loudly rather than producing intermittent hangs. * When a host's networking changes — a new VPN, a move to a tunnelled network — re-check the value; nothing in Docker will tell you it has gone stale. ## What not to say Do not reach for "the application is slow" or "the registry is flaky", do not raise the MTU hoping to push more through, and do not try to change the value on a live network — there is no command that does it.