skip to content

A capture shows VXLAN packets reaching the remote VTEP, yet the tenant hosts never see the frames; which encapsulation-format mismatches would you check?

level: seniorimportance: should knowfreq 12%

answer

  1. the underlay already did its job
  2. which port the receiver decapsulates
  3. the I bit and a known VNI
  4. inner tag kept or stripped
  5. checksum: wrong, or zero on IPv6

basics

~20 s

Walk the packet against RFC 7348: a destination port the receiver does not decapsulate, a clear I bit or unknown VNI, an inner VLAN tag the receiver discards by default, or a UDP checksum it rejects.

solid answer

~50 s

Packets reaching the VTEP means routing works; frames die at **decapsulation**, so walk the format outside in. **Destination port:** `4789` is the default but configurable, since early implementations used other ports - on a mismatch the receiver never treats the datagram as VXLAN. **VXLAN header:** the I bit must be 1 for a valid VNI, and the VNI must be one the receiver serves; reserved bits are ignored, so they are not the cause. **Inner frame:** section 6.1 says a decapsulating VTEP SHOULD discard frames that still carry an inner VLAN tag unless configured otherwise, so a sender that keeps tags meets a receiver that drops them. **Checksum:** a wrong non-zero checksum is dropped by a receiver that verifies, and over IPv6 a zero checksum is discarded unless zero-checksum mode is enabled (RFC 8200, RFC 6936). If only large frames vanish, suspect fragments, which the receiver MAY discard.

go deeper

for a junior

Remember that a packet arriving at the right VTEP can still be dropped there if its port, header or inner frame does not match what that VTEP expects.

for a middle

Know the RFC 7348 rules each check rests on: the configurable port 4789, the I bit, ignoring reserved bits, stripping inner tags, and accepting a zero checksum.

for a senior

Work the capture outside in, separate format faults from size faults, and know that zero checksums behave differently over IPv6 and that the inner-tag rules are configurable defaults on both ends.

for a principal

Turn the failure into prevention: interoperability tests between VTEP implementations for port, tag handling and checksum mode, and alarms on decapsulation drops rather than on reachability alone.

## Read the symptom first The capture is taken at, or just in front of, the receiving **VTEP** (VXLAN Tunnel End Point), and VXLAN packets are arriving. That rules out most of the underlay: routing to the VTEP's address works, and nothing on the path is filtering the traffic. The frames are being lost **at decapsulation** - when the VTEP decides whether this datagram is VXLAN, which segment it belongs to, and whether to hand the inner frame on. Every one of those decisions turns on a field RFC 7348 defines, so the check is a walk down the packet. ## The checklist, layer by layer | Layer | Mismatch to look for | RFC rule | |---|---|---| | Outer IP | destination is not an address this VTEP uses for VXLAN | the outer destination is the receiving VTEP (or a multicast group) | | Outer UDP destination port | not the port the receiver decapsulates | `4789` is the IANA default; RFC 7348 says it SHOULD be configurable for early implementations | | Outer UDP checksum | non-zero and wrong; or zero over IPv6 | a verifying receiver MUST drop a failed checksum; IPv6 needs zero-checksum mode for zero | | VXLAN flags | I bit clear | the I bit MUST be 1 for a valid VNI | | VXLAN VNI | a VNI the receiver has no segment for | the receiving VTEP verifies the VNI before delivery (section 4.1) | | Inner frame | an 802.1Q tag still inside | a decapsulating VTEP SHOULD discard tagged inner frames unless configured otherwise (section 6.1) | ## The likely culprits in more detail - **Port mismatch.** RFC 7348 notes that early implementations used destination ports other than 4789 and therefore keeps the port configurable. Two VTEPs that disagree produce exactly this symptom: packets arrive, but to the receiver they are plain UDP to a port with no VXLAN listener. A host stack may answer them with ICMP port-unreachable messages, which a capture would show. - **VNI mismatch.** The packet is recognised as VXLAN, but its VNI is not one the receiver has a segment for, so there is nowhere to deliver the frame. Compare the VNI in the capture with what each end expects; how VNIs are assigned to segments is a tenancy question, but the field in the header is the evidence. - **I bit clear.** Rare, but a hand-built or buggy encapsulator that writes the flags byte as `0x00` sends a header with no valid VNI. A correct first byte is `0x08`. - **Inner VLAN tag.** Both halves of section 6.1 are SHOULDs "unless configured otherwise": strip the tag when encapsulating, discard tagged frames when decapsulating. One end configured to preserve tags against a peer on the default loses every tagged frame. - **Checksum.** RFC 7348 has senders send zero and receivers accept zero. A sender that computes a non-zero checksum must get it right over the whole packet; a receiver that chooses to verify and finds it wrong MUST drop it. Over **IPv6**, RFC 8200 makes a UDP checksum mandatory by default and has receivers discard zero, unless zero-checksum mode is enabled for that port as RFC 6936 describes - so an IPv6 underlay with zero checksums on one side and no such mode on the other drops everything. ## What is not the cause - **Reserved bits.** RFC 7348 says reserved bits are ignored on receipt, so a non-zero reserved bit is not a reason for a conforming receiver to drop. - **The source port.** It only carries entropy for the underlay; the receiver does not need any particular value. ## When only some frames vanish If small frames get through and large ones do not, look at size rather than format. RFC 7348 says VTEPs **MUST NOT fragment** VXLAN packets, intermediate routers **may** fragment them, and the destination VTEP **MAY silently discard** the fragments. That points to the underlay's MTU, a sizing question of its own. ## A diagnostic order 1. Confirm the outer destination IP and UDP destination port match what the receiver decapsulates. 2. Decode the first 8 bytes after the UDP header: the flags byte should be `0x08`, and the VNI should be one both ends expect. 3. Look at the first bytes of the inner frame for an 802.1Q tag after the source MAC. 4. Check the outer UDP checksum: zero or non-zero, IPv4 or IPv6, and whether the receiver verifies. 5. Compare frame sizes that fail with sizes that work before blaming the format.

  • Why might one VTEP keep the inner 802.1Q tag while its peer discards tagged frames?
    RFC 7348 section 6.1 makes both behaviours SHOULDs "unless configured otherwise": strip the inner tag when encapsulating, discard tagged frames when decapsulating. One end configured to carry tags against a peer left on the default silently loses every tagged frame. The fix is to align both ends; usually that means stripping, since the VNI already identifies the segment.
  • In a capture at the receiving VTEP, how would you tell a VXLAN port mismatch from a VNI mismatch?
    With a port mismatch the packets go to a UDP port the receiver does not treat as VXLAN, and a host stack may answer with ICMP port-unreachable messages. With a VNI mismatch the packets arrive on the right port with the I bit set, but the VNI in bytes 4 to 6 of the VXLAN header is not one the receiver serves.

saying these in an interview costs you the question

  • If the packets reach the VTEP, the problem must be in the tenant host
  • A non-zero reserved bit in the VXLAN header explains the drops
  • A zero UDP checksum makes any VXLAN receiver drop the packet
  • VXLAN carries the inner VLAN tag end to end by default
  • The sending VTEP should fragment large packets so they fit the underlay