skip to content

How are OSPFv3 packets authenticated, given that RFC 5340 removed the authentication fields from the OSPF header?

level: middleimportance: should knowfreq 14%

answer

  1. no AuType in the v3 header
  2. borrow IPv6's own security first
  3. ESP must, AH may
  4. one-to-many breaks key negotiation
  5. a trailer added later

basics

~20 s

OSPFv3 has no authentication of its own. RFC 4552 protects it with IPsec, ESP required and AH optional, using manually configured keys; RFC 7166 adds an HMAC Authentication Trailer appended to each packet, with a 64-bit sequence number against replay.

solid answer

~50 s

RFC 5340 deleted `AuType` and `Authentication` from the OSPF header; the IPv6 upper-layer checksum catches only accidental corruption. The first answer was **IPsec**, in RFC 4552: implementations must support ESP and may support AH, in transport mode. Because OSPF multicasts to many neighbours and IKE agrees keys between two parties only, RFC 4552 mandates **manual keys**, with the same SA for inbound and outbound traffic, and admits that this gives **no replay protection**. The second answer is the **Authentication Trailer**, RFC 7166, which obsoletes RFC 6506: an HMAC digest appended after the OSPFv3 packet and signalled by the `AT-bit` in Hello and Database Description Options, carrying a 16-bit SA ID and a 64-bit **Cryptographic Sequence Number** that rejects replays. HMAC-SHA-256 is mandatory to support and the default, and the IPv6 source address is covered by the digest.

go deeper

for a junior

Remember that OSPFv3 has no password field of its own and is protected either by IPsec or by an authentication trailer added later.

for a middle

Explain RFC 4552's rules, ESP required, AH optional, manual keys, and why manual keys follow from OSPF's one-to-many multicast.

for a senior

Compare the IPsec and trailer options on replay protection, operational burden and what each digest covers, including the IPv6 source address.

for a principal

Decide how routing-protocol authentication is managed across a network: key rotation without outages, mixed implementations, and whether confidentiality of routing data is worth IPsec's cost.

## What RFC 5340 removed An OSPFv2 header carries a 16-bit `AuType` and a 64-bit `Authentication` field. RFC 2328 defines three types: `0` null, `1` simple password and `2` cryptographic authentication with a keyed digest; RFC 5709 later added HMAC-SHA digests for OSPFv2. OSPFv3 drops all of it. RFC 5340 removed `AuType` and `Authentication` from the header and every authentication field from its area and interface data structures, and said OSPF would rely on IPv6's **Authentication Header (AH)** and **Encapsulating Security Payload (ESP)**. What remains inside OSPFv3 is the standard IPv6 upper-layer **checksum** over the packet and a pseudo-header, which catches accidental corruption and authenticates nothing. ## Option 1: IPsec, RFC 4552 RFC 4552 (Standards Track) describes how to protect OSPFv3 with IPsec: - implementations **must support ESP** for authentication and **may support AH**; - **transport-mode** security associations must be supported, and tunnel mode may be; - confidentiality, through ESP, should be supported; - once authentication is enabled, OSPFv3 packets without AH or ESP, or failing their checks, are **silently discarded**. The two protocols cover different bytes. ESP in transport mode authenticates the OSPFv3 packet but not the IPv6 header in front of it; AH covers the packet plus selected parts of the IPv6 header and extension headers. ### Why the keys are manual OSPFv3 multicasts to every router on a link, so the protection it needs is **one-to-many**. IKE is built on Diffie-Hellman key agreement between **two** parties and cannot provide that, so RFC 4552 **mandates manual keying**. Every router on the link uses the same SA parameters, SPI and keys alike, for inbound and outbound traffic. The RFC lists the costs itself: 1. **No replay protection.** Sequence numbers cannot be negotiated, so recorded OSPF packets can be replayed. RFC 4552 warns this can disrupt adjacencies, keep the database exchange looping and cause micro-loops. 2. **Long-lived keys.** Manual keys tend to stay unchanged for a long time, giving an attacker time. 3. **Key changes need a procedure.** Rolling a key requires every router on the link to move through the same steps. ## Option 2: the Authentication Trailer, RFC 7166 RFC 7166 observes that in some environments IPsec proved hard to configure and maintain, and defines a mechanism inside OSPFv3. It obsoletes RFC 6506, which first defined the trailer. | Element | What it does | |---|---| | `AT-bit` in the Options field | set in Hello and Database Description packets to say a trailer is present | | Placement | appended **after** the OSPFv3 packet; counted in the IPv6 payload length, not in the OSPFv3 packet length | | `Authentication Type` and `Auth Data Len` | describe the trailer | | `Security Association ID` | 16-bit number selecting the manually configured key and algorithm | | `Cryptographic Sequence Number` | 64-bit, strictly increasing per packet sent; a packet not greater than the last accepted one of the same type from that neighbour is dropped as a replay | | Authentication data | the HMAC digest | Implementations must support **HMAC-SHA-256**, which is the default; HMAC-SHA-1 is recommended and HMAC-SHA-384 and HMAC-SHA-512 are optional. Two details matter in interviews: - the **IPv6 source address** is fed into the digest, because OSPFv3 copies a neighbour's source address from its Hellos and uses it as a next hop, so an attacker rewriting it could redirect traffic; - when the trailer is used, the OSPFv3 header **checksum is not computed or checked**, as with OSPFv2 cryptographic authentication. ## Comparing the three | | OSPFv2 built-in | OSPFv3 over IPsec | OSPFv3 trailer | |---|---|---|---| | Defined in | RFC 2328, RFC 5709 | RFC 4552 | RFC 7166 | | Where it lives | the OSPF header | an AH or ESP header | after the OSPFv3 packet | | Keys | per-interface configured | manual, shared SA per link | manual, chosen by SA ID | | Replay protection | cryptographic sequence number | none with manual keys | 64-bit sequence number | | Confidentiality | no | possible with ESP | no | ## Mistakes that show up in interviews - Saying OSPFv3 kept `AuType` and added stronger hashes. - Saying IKE negotiates a key per OSPFv3 neighbour. - Making AH the mandatory protocol; RFC 4552 makes ESP mandatory. - Treating the IPv6 checksum as integrity protection against an attacker.

  • Why does OSPFv3 protected by IPsec under RFC 4552 have no replay protection?
    Because the keys are manual. IPsec's anti-replay depends on sequence numbers that peers can negotiate and reset, which manual keying cannot provide, so RFC 4552 states plainly that it offers no replay protection. It warns that replayed OSPF packets can disrupt adjacencies, keep database exchange running and cause micro-loops. RFC 7166's trailer closes the gap with a 64-bit strictly increasing sequence number.
  • Why does RFC 7166 put the IPv6 source address into the OSPFv3 authentication digest?
    OSPFv3 names neighbours by Router ID, but it copies each neighbour's IPv6 source address from its Hellos into the neighbour structure and uses it as a next hop. A man in the middle who rewrote that address could steer traffic without touching the OSPF payload. Including the source address in the digest makes such a rewrite fail verification.

saying these in an interview costs you the question

  • OSPFv3 keeps OSPFv2's AuType field and simply supports stronger hashes.
  • OSPFv3 IPsec keys are negotiated per neighbour with IKE.
  • AH is mandatory for OSPFv3, and ESP is needed only for encryption.
  • The IPv6 checksum authenticates OSPFv3 packets.
  • IPsec-protected OSPFv3 is safe against replayed packets.