What is VXLAN?

VXLAN (Virtual Extensible LAN) carries Ethernet frames inside UDP across a routed IP network, so one layer-2 segment can span many switches.

Defined in RFC 7348, VXLAN wraps each original frame in an 8-byte VXLAN header, then an outer UDP header (destination port 4789), outer IP header and outer Ethernet header. The VXLAN header carries a 24-bit VNI (VXLAN Network Identifier), which names the segment, much as a VLAN ID does, but with about 16 million values instead of 4094. The devices that wrap and unwrap frames are VTEPs (VXLAN Tunnel Endpoints), usually leaf switches or hypervisors, addressed by a loopback in the underlay. The outer UDP source port is normally derived from a hash of the inner frame, so the underlay's ECMP spreads different flows across different paths. With IPv4 outer headers, the encapsulation adds 50 bytes.

The routed network underneath is the underlay, and the stretched segments on top are the overlay. The underlay only needs IP reachability between VTEP loopbacks, typically from OSPF or eBGP in a leaf-spine fabric. VTEPs also need to know which remote VTEP has which MAC address, and how to deliver broadcasts such as ARP. Plain VXLAN does this by flooding and learning, using multicast or static lists of VTEPs; most modern fabrics use EVPN as a BGP control plane instead. The most common mistake is the MTU: hosts send 1500-byte frames, so the underlay must allow at least 1550 bytes, and jumbo frames around 9000 are common. Don't confuse VXLAN, which is only the data plane, with EVPN, which tells VTEPs where things are. The EVPN and VXLAN lesson builds a leaf-spine fabric with FRR and looks at VXLAN packets on the wire.