skip to content

On UDP port 4500, how does an IPsec endpoint tell an IKE message, a UDP-encapsulated ESP packet and a NAT-keepalive apart?

level: middleimportance: should knowfreq 12%

answer

  1. look at the first payload bytes
  2. four zero bytes line up with the SPI
  3. an SPI is never zero
  4. a single byte, 0xFF

basics

~20 s

By the first bytes after the UDP header: four zero bytes, the non-ESP marker, mean IKE; a non-zero first word is an ESP SPI; a one-octet payload of 0xFF is a NAT-keepalive, which only refreshes the NAT mapping.

solid answer

~40 s

RFC 3948 lets IKE and ESP share the single UDP `4500` mapping and makes the payload self-describing. An ESP packet follows the UDP header directly, so its first four bytes are the **SPI**, which MUST NOT be zero. An IKE message is prefixed with the **non-ESP marker**, four zero bytes aligned with where the SPI would be, so a zero first word means IKE. A **NAT-keepalive** is a datagram whose payload is the single octet `0xFF`, too short to hold either; the receiver SHOULD ignore it. On IPv4 the UDP checksum of encapsulated ESP and keepalives SHOULD be sent as zero, because ESP's own integrity check covers the contents.

code

pseudocode · 11 lines
pseudocode
on datagram arriving on UDP port 4500:
    payload = bytes after the 8-byte UDP header
    if length(payload) == 1 and payload[0] == 0xFF:
        discard            # NAT-keepalive: refreshes the NAT mapping only
    else if length(payload) >= 4 and payload[0..3] == 00 00 00 00:
        ike_input(payload[4..])     # strip the non-ESP marker
    else if length(payload) >= 8:
        spi = payload[0..3]         # non-zero by rule
        esp_input(spi, payload)
    else:
        discard            # too short to be ESP

go deeper

for a junior

Recall that port 4500 carries both IKE and ESP after a NAT is found, and that IKE messages there start with four zero bytes.

for a middle

Explain the three formats, why an SPI of zero is forbidden, how the marker lines up with the SPI field, and why the IPv4 UDP checksum may be zero.

for a senior

Read a capture on port 4500 and classify every datagram, and spot an implementation that misroutes keepalives, rejects non-zero checksums or expects the marker before ESP.

for a principal

Judge the shared-port design against separate ports: one mapping and one firewall rule versus a demultiplexing rule every implementation must get exactly right.

## Why one port needs a demultiplexing rule Once IPsec peers detect a NAT, they move both IKE and ESP to **UDP port 4500**. Sharing the port was deliberate: RFC 3948 gives the reasons as better scaling (only one NAT mapping and no separate IKE keepalives), one port to open in firewalls, and simpler implementations. The cost is that every datagram arriving on 4500 could be one of three things, and the receiver must tell which without a separate port. ## The three formats | Datagram | Bytes after the 8-byte UDP header | Defined in | |---|---|---| | UDP-encapsulated ESP | ESP header: SPI (4, never zero), sequence number (4), then ciphertext | RFC 3948 §2.1 | | IKE on port 4500 | **Non-ESP marker** (4 zero bytes), then the IKE header | RFC 3948 §2.2, RFC 7296 §2.23 | | NAT-keepalive | One octet, value `0xFF` | RFC 3948 §2.3 | The receiver's decision is short: 1. If the UDP payload is the single octet `0xFF`, it is a **NAT-keepalive**: ignore it. 2. Otherwise read the first four bytes. If they are all zero, strip them and hand the rest to **IKE**. 3. Otherwise the four bytes are an **SPI**: hand the packet to ESP processing, which looks up the security association. ## Why it is unambiguous - **The SPI can never be zero.** RFC 4303 reserves SPI `0` for local use and forbids sending it on the wire, and RFC 3948 repeats that the SPI of an encapsulated packet MUST NOT be zero. So a zero first word cannot be ESP. - **The marker lines up with the SPI field.** RFC 3948 defines the non-ESP marker as four zero-valued bytes aligned with the SPI field, which is what makes the single comparison work. RFC 7296 states that every IKE message on port 4500 MUST begin with this prefix, or the receiver will not know how to handle it. - **The keepalive is one byte.** It cannot contain a four-byte SPI or marker, so it never looks like either. ## UDP header details - Source and destination ports MUST be the same ones IKE uses, normally `4500`, which after a NAT becomes whatever port the NAT assigned on one side. - For **IPv4**, the UDP checksum of encapsulated ESP and of keepalives SHOULD be transmitted as **zero**, which RFC 768 allows to mean no checksum, and receivers MUST NOT depend on it being zero. ESP's integrity check already protects the contents; a keepalive carries nothing worth checking. - IKE messages on 4500 keep the normal UDP checksum rules; RFC 3948 sets no new requirement for them. - UDP encapsulation MUST NOT be used on port `500`, so the marker never appears there and IKE on 500 has no prefix. - The specification is written in terms of IPv4 but notes that an IPv6 outer or inner header could be used; RFC 3715 points out that IPv6 requires UDP checksums, which is why the zero-checksum allowance is stated for IPv4. ## What the encapsulation adds to a packet For ESP, UDP encapsulation inserts exactly the **8-byte UDP header**, nothing else. In transport mode the IP header's Total Length, Protocol (now UDP, `17`) and header checksum are edited to fit; in tunnel mode the same three fields of the new outer header are. On receipt the UDP header is removed, those fields are restored, and ordinary ESP decapsulation runs. For IKE the overhead on 4500 is the UDP header plus the 4-byte marker. ## Mistakes that show up in captures and code - Treating the keepalive as a liveness signal: RFC 3948 says reception of keepalives MUST NOT be used to decide whether the connection is alive. - Expecting the marker in front of ESP: it is only in front of IKE. - Rejecting encapsulated ESP that arrives with a non-zero UDP checksum: receivers must not depend on zero.

  • Why does IKE on port 500 not carry the non-ESP marker?
    Port `500` never carries encapsulated ESP; RFC 7296 forbids UDP encapsulation there. With nothing to distinguish, IKE on 500 starts directly with its header, whose first field is the initiator's SPI. The marker exists only because port `4500` multiplexes IKE with ESP.
  • Why is it acceptable for UDP-encapsulated ESP to skip the IPv4 UDP checksum?
    ESP's integrity check value already authenticates everything from the SPI onward with a cryptographic key, which is far stronger than a 16-bit checksum. RFC 3948 therefore says the IPv4 UDP checksum SHOULD be zero, while receivers MUST NOT depend on it being zero.

saying these in an interview costs you the question

  • The non-ESP marker is prepended to every ESP packet on port 4500.
  • An ESP SPI of zero marks a NAT-keepalive.
  • Receiving NAT-keepalives proves the peer is still alive.
  • UDP-encapsulated ESP must carry a valid UDP checksum on IPv4.
  • IKE and ESP use separate UDP ports once a NAT is detected.