skip to content

What can an underlay router see of VXLAN traffic between two VTEPs, and how does that shape underlay filtering and troubleshooting?

level: seniorimportance: should knowfreq 18%

answer

  1. forwarding stops at the outer headers
  2. VTEP pairs, not tenant flows
  3. port 4789 and a hashed source port
  4. cleartext, but not routed on
  5. start debugging at the VTEP addresses

basics

~20 s

An underlay router forwards on the outer headers: VTEP source and destination addresses, UDP port 4789 and a hashed source port. Tenant addresses sit inside the payload, so outer-header filters act per VTEP pair, and tenant-level debugging starts at the VTEPs.

solid answer

~50 s

An underlay router sees an outer Ethernet header, an outer IP header from one VTEP address to another with protocol 17, and a UDP header with destination port `4789` and a source port the sending VTEP hashes from the inner headers. The VNI and the tenant's frame are UDP payload: VXLAN does not encrypt them, so a capture can decode them, but routers forwarding on outer headers never use them. Two consequences follow. **Filtering** works at VTEP granularity: permit UDP `4789` only between VTEP addresses and drop VXLAN from anywhere else, because RFC 7348 notes that anyone able to inject MAC-over-UDP into the transport network can inject frames; per-tenant policy belongs at the VTEPs. **Troubleshooting** inverts: tenant traceroutes show no underlay hops and flow records show VTEP pairs, so first prove VTEP-to-VTEP reachability, then follow a specific flow by its outer five-tuple.

go deeper

for a junior

Recall that the underlay sees VTEP addresses and UDP port 4789, while the tenant's own addresses travel hidden inside the packet.

for a middle

Explain each layer an underlay router reads and why it ignores the VNI and inner frame, including the hashed UDP source port.

for a senior

Show the production habits: coarse VTEP-to-VTEP edge filters, tenant policy at the VTEP, and a debugging order that proves VTEP reachability before chasing a single flow.

for a principal

Weigh the visibility gap as a design cost: decide where tenant-aware inspection lives, whether VTEPs must export flow data, and what the operations team can see.

## What is on the wire Between two **VTEPs**, a VXLAN packet is an ordinary UDP/IP packet whose payload happens to be an Ethernet frame. An underlay router reads it from the outside in and stops as soon as it has what it needs to forward. | Layer | Contents | Used by underlay forwarding? | |---|---|---| | Outer Ethernet | next-hop MAC addresses | yes, rewritten per hop | | Outer IP | source and destination VTEP addresses, protocol 17, TTL, DSCP | yes | | UDP | source port hashed from inner headers, destination port `4789` | for load balancing and filters | | VXLAN header | flags and the 24-bit VNI | no | | Inner frame | tenant MAC and IP addresses, tenant payload | no | RFC 7348 recommends that the sending VTEP compute the UDP source port from a hash of the inner packet's headers, so different tenant flows between the same two VTEPs carry different outer source ports. The VNI and the inner frame are not encrypted by VXLAN itself, so a packet capture that decodes port `4789` can display them; the point is that a router forwarding on outer headers has no reason to look. ## What the underlay cannot use - **Tenant addresses.** A route or an access list in the underlay matches outer addresses, which are VTEP addresses; the tenant's `192.168.10.0/24` never appears there. - **Segment identity.** To a router forwarding on outer headers, two tenants' traffic between the same pair of VTEPs differs only in the hashed source port. - **Tenant hop counts.** Within a segment the VTEPs bridge, and RFC 8014 says the tenant packet's TTL or Hop Limit is not modified, so a tenant's traceroute never expires inside the underlay. ## Filtering consequences - **Underlay filters work at VTEP granularity.** RFC 7348 points out that the UDP encapsulation lets physical switches apply five-tuple access lists, and the useful rule is coarse: allow UDP to port `4789` between known VTEP addresses, drop it from everything else. - **Injection is the risk being filtered.** RFC 7348's security section warns that rogues could source MAC-over-UDP frames into the transport network to inject spurious traffic or hijack MAC addresses. VXLAN carries no authentication of its own, so keeping VTEP addresses unreachable from outside the fabric and dropping foreign `4789` traffic at its edge does that job. - **Tenant policy belongs at the VTEP** or at a device that terminates the overlay, because only there are the tenant's addresses and segment visible as forwarding keys. Where traffic must cross untrusted networks, RFC 7348 names IPsec as the means to authenticate and encrypt it. - **Keep VXLAN on its own segment where VTEPs share a LAN.** RFC 7348 also recommends designating a VLAN for VXLAN traffic, with servers and VTEPs sending it only over that VLAN, as a further measure. ## Troubleshooting consequences The underlay's view turns tenant complaints into questions about VTEP pairs. A workable order: 1. **Map the tenant flow to its VTEPs.** Find which VTEP serves each end: the source host's VTEP and the remote VTEP its MAC-to-VTEP table points to. 2. **Prove VTEP-to-VTEP reachability** in both directions, sourcing tests from the VTEP addresses themselves, since that is what the encapsulated traffic uses. 3. **Follow the specific flow.** Underlay counters and flow records show VTEP pairs, with many tenant flows behind each; the outer source port is what distinguishes them, and where the underlay hashes on UDP ports it also decides which equal-cost path a flow takes, so reproduce or capture that five-tuple rather than a generic ping. 4. **Capture at both VTEPs.** If the packet leaves one VTEP encapsulated and never arrives at the other, the underlay is at fault; if it arrives and is not delivered, look at the receiving VTEP's VNI and MAC checks. The division is clean because the two sides share nothing but the outer header: the underlay team can prove its part with VTEP addresses alone, and the overlay team can prove theirs inside the VTEPs. ## Quality of service The underlay schedules on the outer header too. RFC 7348 does not say how the outer DSCP is set, so whether a VTEP copies the tenant packet's marking into the outer IP header is an implementation choice, and one worth checking when tenant traffic loses its priority as soon as it enters the tunnel.

  • Why is one slow tenant flow hard to find in the underlay's flow records for VXLAN traffic?
    The underlay records outer five-tuples: VTEP source and destination addresses, UDP, destination port 4789 and the hashed source port. Every tenant flow between the same two VTEPs shares the addresses and differs only in that source port, so mapping a record to a tenant flow needs the VTEP's hash or a capture that decodes the inner headers.
  • How would you keep forged VXLAN packets from outside the fabric away from your VTEPs?
    VXLAN has no authentication of its own; RFC 7348 relies on other mechanisms. Keep VTEP addresses out of external routing so outsiders cannot reach them, and at the fabric edge drop UDP to port 4789 unless it comes from the VTEP address range. Where traffic crosses untrusted networks, protect it with IPsec, as RFC 7348 suggests.

saying these in an interview costs you the question

  • An underlay ACL on IP addresses can block one tenant VM, because it matches the VM's address.
  • VXLAN encrypts the inner frame, so an underlay capture shows nothing useful.
  • A tenant traceroute will reveal which spine its VXLAN traffic crossed.
  • A VTEP accepts VXLAN packets only from VTEPs it has authenticated.
  • All flows between two VTEPs take one underlay path because the outer addresses match.