IPsec

IKE authenticates the peers, negotiates algorithms and derives keys for ESP or AH. IKEv1 split this into Phase 1 (Main or Aggressive Mode, building a protected IKE SA) and Phase 2 (Quick Mode, negotiating IPsec SAs under it); RFC 9395 deprecates IKEv1.

IPsec transport mode keeps the original IP header and inserts ESP between it and the payload, protecting only that payload; tunnel mode puts the whole original packet, header included, behind ESP and a new outer IP header.

ESP is IP protocol 50 with no port numbers, so a port-translating NAT cannot tell which inside host an inbound packet is for. NAT traversal puts a UDP header on port 4500 in front of ESP, giving the NAT ports to map.

An IPsec Security Association is one-way state: the keys, algorithms, counters and selectors protecting traffic in a single direction with AH or ESP. Two-way traffic therefore needs two SAs, one per direction, each with its own SPI.

AH's integrity check covers the IP source and destination addresses, which a NAT rewrites, so the receiver's recomputed ICV no longer matches and the packet is dropped. ESP's check excludes the outer header, so only ESP can be repaired.

IKE_SA_INIT negotiates the IKE SA's algorithms and exchanges Diffie-Hellman values and nonces in the clear, because no keys exist yet. IKE_AUTH, encrypted under the derived keys, carries identities, AUTH proofs, and the first Child SA's proposal and traffic selectors.

Each IKEv2 peer sends SHA-1 hashes of the SPIs with its source and destination address and port in IKE_SA_INIT. A mismatch reveals a translator and which side is behind it; IKE and ESP then move to UDP 4500.

In a policy-based IPsec VPN, Security Policy Database selectors decide which packets are encrypted, typically one selector pair per protected subnet pair. In a route-based VPN, a virtual tunnel interface carries wide selectors and the routing table decides what enters it.

With AES-GCM the IPsec tunnel-mode packet would be 1,556 bytes, too big, so the gateway SHOULD drop it and send an ICMP PMTU message to the source; the largest inner packet that fits is 1,446 bytes.

Ten bytes: AES-CBC needs payload, padding, Pad Length and Next Header to fill whole 16-byte blocks, so 100 + 2 rounds up to 112. Pad Length records 10; Next Header names the payload: 6 for TCP, 4 for IPv4.

When nothing protected has arrived recently, an IKEv2 gateway sends an empty INFORMATIONAL request and retransmits it with exponential backoff; only repeated silence, or a protected INITIAL_CONTACT on a new IKE SA, proves failure. ICMP and unprotected notifies are forgeable.

Both compute AUTH over the sender's IKE_SA_INIT message, the peer's nonce and its own ID. A pre-shared key makes AUTH a keyed prf, guessable offline if the secret is weak; a signature uses a private key a certificate binds to the ID.

In IPsec transport mode an observer reads the real source and destination hosts from the original IP header; tunnel mode shows only the two gateways. Both encrypt ports and the next-layer protocol; sizes, timing and, by default, DS marks still leak.

With IPsec ESP and AES-GCM (8-byte IV, 16-byte ICV), the 1,400-byte IPv4 packet becomes 1,436 bytes in transport mode (36 bytes added) and 1,456 bytes in tunnel mode (56 added), the difference being the outer IPv4 header.

By the first bytes after the UDP header: four zero bytes, the non-ESP marker, mean IKE; a non-zero first word is an ESP SPI; a one-octet payload of 0xFF is a NAT-keepalive, which only refreshes the NAT mapping.

The SPD is an ordered list of entries keyed by selectors such as addresses, protocol and ports, each saying PROTECT, BYPASS or DISCARD. Overlapping ranges make order decide which entry applies, and traffic matching nothing is discarded.

The receiver looks up the packet's 32-bit SPI, which it chose itself, in its Security Association Database; no match means discard. It applies ESP with that entry's keys, then checks the inner headers against the SA's selectors.

The client omits AUTH from its first IKE_AUTH message; the gateway answers with its certificate, signed AUTH and an EAP request; the method runs over extra IKE_AUTH round trips; both sides finish with AUTH keyed from the EAP MSK.

Sequence numbers are assigned at encryption; priority queuing then lets later numbers overtake. The receiver's window, 64 by default, moves past the delayed packets, which fall left of it and are dropped. Fix: one SA per traffic class, or a larger window.

The first Child SA, built in IKE_AUTH, uses no Diffie-Hellman group; a Child SA group is first offered at the CREATE_CHILD_SA rekey. If the peers disagree that fails, RFC 7296 makes the endpoint close the IKE SA, and a fresh setup succeeds again.

Locate the failure by exchange: NO_PROPOSAL_CHOSEN in IKE_SA_INIT is an algorithm mismatch, AUTHENTICATION_FAILED in IKE_AUTH a secret, identity or certificate mismatch, TS_UNACCEPTABLE a selector mismatch with the IKE SA up, and an unanswered IKE_AUTH often dropped fragments.

RFC 4301 lets a router use IPsec transport mode for packets it sources itself, and GRE packets carry the router's own address; tunnel mode would add a second IPv4 header with the same two addresses, 20 redundant bytes.

Each direction is its own SA, so one can fail alone. Usually B's replies never match its PROTECT entry, or A drops them: their SPI is unknown to A, or their inner headers fall outside the SA's selectors.

With one Child SA pair per subnet pair, policy-based tunnels need 100 pairs per branch: 4,000 pairs or 8,000 one-way ESP SAs at the hub. Route-based tunnels with any-to-any selectors need one pair per tunnel: 40 pairs, 80 SAs.

The peers' selector configurations disagree. The initiator proposes a wide range led by the triggering packet's addresses; a narrower responder MUST keep a subset containing that first selector, so only the subnet that sent first gets a Child SA.

Build two route-based tunnels per branch with any-to-any selectors, run BGP over each, and let route withdrawal move traffic. Then decide active/standby versus active/active, how fast failure is detected, and how routes are filtered.

With anti-replay on, an ESP sender MUST NOT let the sequence number cycle: an SA carries at most 2^32 − 1 packets, about 72 minutes at 1 Mpps. Extended sequence numbers count in 64 bits but send only the low 32.

With MOBIKE the client keeps its IKE SA and Child SAs: it sends an UPDATE_SA_ADDRESSES notify from the new address, the gateway checks that address, and only the outer tunnel addresses change while inner addresses and selectors stay.

ESP with ENCR_NULL and an integrity transform authenticates the payload, crosses NAT and is mandatory to implement; AH is optional and fails behind NAT. The cost: no outer-header integrity, and nothing in the packet shows it is unencrypted.