Why does a hardware VXLAN VTEP usually source its tunnels from a loopback address, and what breaks when that address is missing from underlay routing?
answer
- the VTEP address is the tunnel
- survives any single uplink
- a host route in the underlay
- outbound works, return does not
- test from the loopback itself
basics
~20 sA loopback stays reachable while any uplink survives, so it makes a stable VTEP address. It is every tunnel's outer source and destination; if the underlay lacks its route, return traffic to that VTEP is dropped while its outbound traffic still flows.
solid answer
~50 sA VTEP's address is the outer source IP on everything it sends and the outer destination IP every other VTEP uses to reach it. On a leaf switch with two uplinks, tying that address to one uplink would make it vanish with that link; a loopback belongs to no physical interface, so once the underlay routing protocol advertises it as a host route, it stays reachable over whichever uplinks survive, with equal-cost paths across both. RFC 7348 does not mandate this; it is a design convention. If the loopback is missing from underlay routing, the failure is one-way: this VTEP's packets still reach remote VTEPs, whose loopbacks are routed, and are delivered, but replies addressed to its loopback are dropped. Tenants see requests arrive and answers vanish. Test by pinging loopback to loopback, sourced from the loopback, in both directions, and check that each spine holds the host route.
go deeper
Recall that every VTEP has one underlay IP address, usually on a loopback, that peers use to reach it.
Explain why the loopback beats an uplink address: independence from any single link, a host route in the underlay, and equal-cost reachability.
Recognise the one-way failure of an unadvertised VTEP address and prove it with loopback-sourced tests in both directions and per-spine route checks.
Treat VTEP addressing as part of the fabric's addressing plan: allocation, advertisement, protection from outside reach, and where shared addresses for dual-homed servers are acceptable.
## The VTEP address is the tunnel VXLAN tunnels are stateless, so there is nothing to a tunnel except the address on each end. RFC 7348 sets the roles: the **outer source IP** of every packet is the address of the VTEP serving the sending host, and when the **outer destination** is unicast it is the address of the VTEP serving the destination MAC. Every remote VTEP that holds a MAC-to-VTEP mapping for hosts behind this switch stores this one address. If it becomes unreachable, every tunnel to this VTEP fails at once, and if it changes, they fail until every peer maps the hosts to the new address. ## Why a loopback A leaf switch with a hardware VTEP typically has two or more routed uplinks to the spines. Its VTEP needs one address that does not depend on any of them. - **Independent of interfaces.** A loopback is a logical interface that stays up while the switch is up; an uplink's address disappears with that link. - **Reachable over every path.** Advertised as a host route (`/32` in IPv4) into the underlay, the loopback is reachable through every surviving uplink, and equal-cost routing can spread tunnel traffic across all of them. - **Stable for the peers.** Remote VTEPs keep one address per VTEP whatever happens to individual links. - **A convention, not a protocol rule.** RFC 7348 says nothing about loopbacks; it requires only that the outer addresses be the VTEPs'. RFC 7938, describing routed data-centre fabrics, treats a device's loopback as its primary address, always advertised into BGP. ## Getting the address into the underlay 1. Assign the VTEP address to a loopback, unique to this VTEP, for example `10.0.0.11/32`. 2. Advertise it into the underlay's routing protocol, whether an IGP or BGP, so every leaf and spine learns a route to it. 3. Configure the VTEP to use that loopback as the source of its encapsulated packets, so the outer source address matches the address peers will send to. 4. Make sure remote VTEPs map this switch's hosts to that same address, whether they learn it from traffic or from a control protocol. ## The failure: the loopback is not routed Suppose leaf 1's VTEP `10.0.0.11` is configured and used as the source, but step 2 was missed. Leaf 2's VTEP is `10.0.0.12` and correctly advertised. | Direction | What happens | |---|---| | Leaf 1 to leaf 2 | outer destination `10.0.0.12` is routed; leaf 2 decapsulates and delivers | | Leaf 2 to leaf 1 | outer destination `10.0.0.11` has no route; the underlay drops it, or sends it wherever a default route points | | Tenant view | requests from rack 1 reach rack 2; the replies never come back | The asymmetry is the signature. A one-way failure between two VTEPs points at the reachability of one VTEP address, not at the tenant configuration. It also fools tenant-side checks: the remote host does receive traffic, its counters move, and its replies are sent, so host-level checks on the remote side look healthy and only the wait for the answer fails. ## Diagnosing it 1. **Ping loopback to loopback, sourced from the loopback**, in both directions. A ping sourced from an uplink address can succeed, because its reply returns to the uplink address, which is routed. 2. **Check the host route** for each VTEP address on every spine; a missing route on one spine can break only the flows hashed onto that spine. 3. **Compare the configured source with the advertised address**; a VTEP that sources from one address while peers send to another fails the same way. 4. **Capture at the remote VTEP.** If its replies leave encapsulated toward `10.0.0.11` and never arrive, the underlay is dropping them. ## Other addressing rules - **Unique per VTEP**, unless an implementation deliberately shares one address between a pair of switches serving dual-homed servers; that coordination is a multihoming mechanism of its own. - **IPv4 or IPv6.** RFC 7348 defines both outer header forms, so an IPv6 underlay works the same way with an IPv6 loopback. - **Separate from tenant space.** VTEP addresses live in the underlay's routing table; tenant subnets do not need to appear there. - **More than one address is possible.** RFC 8014 notes an NVE may be multihomed with several underlay addresses, one per interface, and then senders need one-to-many mappings and failover between those addresses; a single routed loopback avoids that complexity.
- Why can a ping between two VXLAN VTEP switches succeed while tenant traffic between them fails?The ping may be sourced from an uplink address, whose reply comes back to a routed interface address, while VXLAN return traffic is addressed to the loopback the VTEP uses as its source. If that loopback is not routed, the ping proves nothing about it. Always source the test from the VTEP address. Small pings can also pass a path that large encapsulated frames cannot.
- Can one VXLAN VTEP have more than one underlay address?Yes. RFC 8014 says an NVE may be multihomed, with several underlay IP addresses, typically one per interface, and that a tenant system may also be reachable through more than one NVE. Senders then need one-to-many mappings and the ability to fail over between addresses. A single loopback reachable over every uplink sidesteps that.
saying these in an interview costs you the question
- A VTEP should use its uplink interface address so the tunnel follows that link.
- If remote VTEPs reach me, I reach them, so testing one direction is enough.
- Tenant subnets must be advertised into the underlay alongside the VTEP loopbacks.
- RFC 7348 requires every VTEP to source its tunnels from a loopback interface.
- Any two VTEPs can reuse one loopback address without further coordination.