Why does plain IPsec ESP often fail through a home router's port-translating NAT, and what does NAT traversal change on the wire?
answer
- what a NAPT multiplexes on
- protocol 50 has no ports
- SPIs chosen independently by each receiver
- wrap ESP in UDP 4500
basics
~20 sESP is IP protocol 50 with no port numbers, so a port-translating NAT cannot tell which inside host an inbound packet is for. NAT traversal puts a UDP header on port 4500 in front of ESP, giving the NAT ports to map.
solid answer
~50 sA home router doing NAPT shares one public address by rewriting **ports**, and it finds the inside host for a reply by looking the port up. ESP (IP protocol `50`) has no ports: after the outer IP header comes the SPI and the encrypted payload. The NAT can rewrite the source address, but with two clients talking to the same gateway it has no reliable key for the return traffic, and the inbound SPI is chosen by the receiver, so the NAT cannot learn it from outbound packets. IPsec NAT traversal (RFC 3948 with IKEv2, RFC 7296 §2.23) fixes this by inserting an ordinary 8-byte UDP header, ports `4500`/`4500`, in front of the unchanged ESP header. The NAT now sees a normal UDP flow it can translate, and both peers send IKE and ESP over that one mapping.
go deeper
Recall that ESP is IP protocol 50 with no ports, that a NAPT router needs ports to tell inside hosts apart, and that NAT traversal wraps ESP in UDP 4500.
Explain the RFC 3715 failure list: no ports, independently chosen SPIs, IKE's fixed port 500. Then walk through detection in IKE_SA_INIT, the move to 4500 and the 8-byte UDP header.
Show what it means operationally: firewall rules for UDP 500 and 4500, the extra 8 bytes against the MTU, why two clients behind one NAT break plain ESP, and why the peer behind the NAT must keep the mapping warm.
Weigh when to force UDP encapsulation everywhere against using it only when a NAT is detected, and when a TCP fallback is worth its performance cost for users on networks that filter UDP.
## What a port-translating NAT needs A home router usually performs **NAPT** (network address and port translation): many private hosts such as `192.168.1.10` and `192.168.1.11` share one public address such as `198.51.100.7`. Outbound, it rewrites the source address and, when needed, the source **port**, and records the pair in a mapping table. Inbound, it looks up the destination port of the reply to find which inside host should receive it. Everything depends on the transport header having ports. ## Why plain ESP has nothing to offer it **ESP** (Encapsulating Security Payload, RFC 4303) is not carried in TCP or UDP. The IP header's Protocol field says `50`, and right after it come: - the **SPI** (Security Parameters Index, 4 bytes), naming the receiver's security association; - the **sequence number** (4 bytes); - the encrypted payload, the trailer and the integrity check value. There is no port anywhere a NAT can read. RFC 3715, the IPsec-NAT compatibility requirements, lists what goes wrong: | Problem | Why it hurts plain ESP | |---|---| | No ports to multiplex | Several inside hosts reaching the same gateway look identical to the NAT on the return path | | SPI chosen by the receiver | Inbound and outbound SPIs are picked independently, so the NAT cannot learn the inbound SPI from outbound traffic; two inside hosts can even pick the same value | | Fixed IKE port | IKE starts on UDP `500`; NATs that special-case port 500 or do not translate it collide when two clients connect | | Transport-mode checksums | The TCP/UDP checksum covers the IP addresses but sits encrypted inside ESP, so the NAT cannot fix it | A single client behind a NAT that tracks ESP by address may get lucky; two clients to the same gateway, or a NAT that drops protocol 50, end the luck. (AH is worse still, because it authenticates the outer addresses themselves, and NAT traversal does not cover it.) ## What NAT traversal changes IPsec NAT traversal is two cooperating specifications: **NAT detection** inside IKEv2 (RFC 7296 §2.23; for the deprecated IKEv1 it was RFC 3947) and **UDP encapsulation of ESP** (RFC 3948). The sequence is: 1. Both peers put `NAT_DETECTION_SOURCE_IP` and `NAT_DETECTION_DESTINATION_IP` notifications in `IKE_SA_INIT`: hashes of the addresses and ports each believes are in use. 2. If a hash does not match what arrived, a translator rewrote the packet, and the peers know which side sits behind it. 3. The initiator moves all further IKE and ESP for that IKE SA to UDP port `4500`; the responder replies to whatever source port arrives, because the NAT has rewritten it. 4. ESP is sent with a standard 8-byte **UDP header** inserted between the outer IP header and the ESP header. The ESP packet itself is unchanged. 5. The peer behind the NAT sends one-byte **NAT-keepalives** when it has nothing else to send, so the mapping does not expire. On the wire, a tunnel-mode packet goes from `IP | ESP | inner packet | trailer | ICV` to `IP | UDP 4500 | ESP | inner packet | trailer | ICV`. The NAT translates that UDP flow like any other. ## What it costs and what it needs - **8 bytes per packet** for the UDP header, on top of ESP's own overhead, which lowers the usable MTU slightly. - Firewalls on the path must permit **UDP 500 and UDP 4500**; plain protocol 50 is no longer needed once encapsulation is in use. - IKE and ESP now **share one port and one mapping**, so the receiver must separate them: IKE messages on 4500 begin with four zero bytes, the **non-ESP marker**, and an ESP SPI is never zero. - RFC 7296 makes NAT traversal optional to implement, but once a NAT is detected both peers MUST encapsulate ESP in UDP. ## How it is often misdescribed - It is not a different cipher or a different ESP: the ESP header, encryption and integrity check are exactly what they would be without a NAT. - It is not a port-forwarding rule on the home router: the client opens the mapping outbound and keeps it open. - It is not a fix for AH, and it is not STUN-style hole punching; it works because one peer, usually the gateway, has a reachable public address.
- What if the network blocks UDP entirely, so even UDP 4500 cannot pass?Then UDP encapsulation cannot help. RFC 9329, which obsoletes RFC 8229, defines carrying IKE and ESP inside a TCP stream that starts with the `IKETCP` prefix; implementations MUST support TCP port `4500`, and a preconfigured alternate port is allowed. It is meant as a fallback because TCP-in-TCP performs worse, and support is preconfigured rather than negotiated.
- Can a peer use UDP 4500 from the very first IKE message?Yes. RFC 7296 lets an initiator use port `4500` for IKE and ESP even when no NAT exists, and every implementation that supports NAT traversal must accept UDP-encapsulated ESP at any time. Starting on 4500 skips the float and avoids NATs that treat port `500` specially.
- Why does NAT traversal leave AH out?AH's integrity check covers the outer IP addresses, so any translation invalidates it whether or not a UDP header is added. RFC 3948 says AH was left out of scope for exactly that reason; ESP's check does not cover the outer header, which is what makes encapsulation sufficient.
Plain ESP is a parcel with a street address but no flat number, delivered to a block of flats with one letterbox: the concierge cannot tell which tenant it is for. NAT traversal writes a flat number on the outside, the UDP port, without opening the parcel.
saying these in an interview costs you the question
- ESP fails behind NAT because it encrypts the outer IP header.
- NAT traversal re-encrypts ESP into a different, NAT-safe format.
- UDP-encapsulated ESP is sent on UDP port 500 alongside IKE.
- The home router needs a port-forwarding rule for NAT traversal to work.
- NAT traversal also makes AH work through a translator.