TTR Networks

Intermediate · BFD on VyOS: fast failover for quiet failures · section 7 of 7

Build the lab yourself

✓ Lab-tested 21 October 2026 on VyOS 2026.10.16-0024-rolling, FRR 10.2.1

The lab is one VyOS VM, one FRR container and two hosts. It needs about 1.5 GB of free memory for Docker.

  1. Get the VyOS image as in the tcpdump lesson (get-vyos.sh), into ./vyos.
  2. Put lab-compose.yaml, start-vyos.sh, hosts-setup.sh, fail-test.sh and r2-daemons (FRR's daemons file with bfdd=yes) in one directory. Copy r2-ospf-bfd.conf to r2.conf; for the first test, delete its four ip ospf bfd lines.
  3. docker compose up -d --build, then ./hosts-setup.sh.
  4. When VyOS has booted, paste R1's stages in order, from the console.
  5. Run ./fail-test.sh NAME SECONDS for each measurement: it starts the pings, cuts link A five seconds in, and puts it back at the end. Then enable BFD on both routers and run it again.
  6. For the BGP part, replace r2.conf with r2-bgp-bfd.conf (without its bfd lines for the first run) and docker compose restart r2; then R1's BGP stages.

On R2, vtysh gives FRR's shell: vtysh -c 'show bfd peers'.

lab-compose.yaml
# Needs ./vyos (from the tcpdump-and-vyos lesson's get-vyos.sh). Then: docker compose up -d --build && ./hosts-setup.sh
# VyOS console: docker compose exec -it r1 socat -,raw,echo=0,escape=0x1d UNIX-CONNECT:/run/console.sock
# R2's config: copy r2-ospf-bfd.conf (or r2-bgp-bfd.conf) to ./r2.conf first.
name: bfd-lab
x-host: &host
  build:
    dockerfile_inline: |
      FROM debian:bookworm-slim
      RUN apt-get update && apt-get install -y --no-install-recommends iproute2 iputils-ping procps
      CMD ["sleep", "infinity"]
  image: ttr-bfd-host
  privileged: true
services:
  r1:
    build:
      context: .
      dockerfile_inline: |
        FROM debian:bookworm-slim
        RUN apt-get update && apt-get install -y --no-install-recommends qemu-system-x86 iproute2 socat
        COPY start-vyos.sh /usr/local/bin/start-vyos.sh
        CMD ["/usr/local/bin/start-vyos.sh"]
    cap_add: [NET_ADMIN]
    devices: ["/dev/net/tun:/dev/net/tun"]
    environment: { VYOS_MEM: "1536", VYOS_SMP: "1", VM_ID: "21" }
    volumes: ["./vyos:/vyos:ro"]
    networks: { linka: { interface_name: eth0 }, linkb: { interface_name: eth1 }, lan1: { interface_name: eth2 } }
  r2:
    image: quay.io/frrouting/frr:10.2.1
    hostname: R2
    privileged: true
    volumes: ["./r2-daemons:/etc/frr/daemons:ro", "./r2.conf:/etc/frr/frr.conf"]
    networks: { linka: { interface_name: eth0, priority: 100 }, linkb: { interface_name: eth1, priority: 90 }, lan2: { interface_name: eth2 } }
  h1: { <<: *host, hostname: h1, networks: { lan1: { interface_name: eth0 } } }
  h2: { <<: *host, hostname: h2, networks: { lan2: { interface_name: eth0 } } }
# Layer-2-only networks: every address comes from the router configs and hosts-setup.sh.
networks:
  linka: { enable_ipv4: false, enable_ipv6: true, ipam: { config: [{ subnet: "fd00:b5::/64" }] } }
  linkb: { enable_ipv4: false, enable_ipv6: true, ipam: { config: [{ subnet: "fd00:b6::/64" }] } }
  lan1:  { enable_ipv4: false, enable_ipv6: true, ipam: { config: [{ subnet: "fd00:b7::/64" }] } }
  lan2:  { enable_ipv4: false, enable_ipv6: true, ipam: { config: [{ subnet: "fd00:b8::/64" }] } }
hosts-setup.sh
#!/bin/sh
# Address the two LAN hosts, and let R2 (FRR) forward.
cd "$(dirname "$0")"
x() { docker compose exec -T "$@"; }
x r2 sh -c 'sysctl -qw net.ipv4.ip_forward=1 net.ipv6.conf.all.disable_ipv6=1'
x h1 sh -c 'sysctl -qw net.ipv6.conf.all.disable_ipv6=1; ip addr add 198.51.100.10/25 dev eth0; ip route add default via 198.51.100.1'
x h2 sh -c 'sysctl -qw net.ipv6.conf.all.disable_ipv6=1; ip addr add 198.51.100.130/25 dev eth0; ip route add default via 198.51.100.129'
echo hosts ready
start-vyos.sh
#!/bin/sh
# Bridge every container interface (eth0, eth1, …) to a tap, then boot VyOS with one virtio NIC per tap.
# VM NIC n gets MAC 52:54:00:<VM_ID>:0n:01, so VyOS's ethN matches the container's ethN. Give each VM its own VM_ID.
set -e
NETDEVS=""
n=0
for ifc in $(ls /sys/class/net | grep -E '^eth[0-9]+$' | sort -V); do
  ip link add "br$n" type bridge
  ip link set "$ifc" master "br$n"
  ip tuntap add "tap$n" mode tap
  ip link set "tap$n" master "br$n"
  ip link set "tap$n" up
  ip link set "br$n" up
  # Pin the VM's MAC to its tap and stop learning on the Docker side: Docker reflects the VM's
  # broadcasts back, which would otherwise teach the bridge the VM's MAC on the wrong port.
  bridge link set dev "$ifc" learning off
  bridge fdb replace "52:54:00:${VM_ID:-00}:0$n:01" dev "tap$n" master static
  NETDEVS="$NETDEVS -netdev tap,id=n$n,ifname=tap$n,script=no,downscript=no -device virtio-net-pci,netdev=n$n,mac=52:54:00:${VM_ID:-00}:0$n:01"
  n=$((n + 1))
done
# VYOS_TB_MB caps QEMU's TCG translation cache (QEMU's default is 1 GB on top of the guest RAM).
exec qemu-system-x86_64 -m "${VYOS_MEM:-2048}" -smp "${VYOS_SMP:-2}" -cpu max -accel tcg,thread=multi,tb-size=${VYOS_TB_MB:-256} \
  -kernel /vyos/vmlinuz -initrd /vyos/initrd.img \
  -append "boot=live components hostname=vyos username=live nopersistence noautologin nonetworking union=overlay console=ttyS0,115200 net.ifnames=0 biosdevname=0" \
  -cdrom /vyos/vyos.iso $NETDEVS \
  -display none -serial unix:/run/console.sock,server=on,wait=off -monitor none

Where next

  • BGP on loopbacks vs interface addressesAlso covers BFD, OSPF, BGP · Intermediate · 45 min

    Why BGP sessions between loopbacks survive link failures and use parallel links, and what they need to work: an IGP or static routes, update-source, the eBGP connected check, and BFD. Every failure is reproduced on two real VyOS routers.

  • OSPF on VyOS: costs and OSPFv3Prerequisite · Intermediate · 30 min

    OSPF the VyOS way: interface-based configuration, reading the interface and database output, steering traffic with costs (and the asymmetric routing that follows), and OSPFv3 for IPv6 over link-local addresses. Captured on the three-router edge/core lab.

  • BGP FundamentalsPrerequisite · Intermediate · 75 min

    How the Border Gateway Protocol glues the internet together: autonomous systems, sessions, path attributes, best-path selection, iBGP scaling and a hands-on eBGP lab with real FRR configuration.

  • Edge and core: building out a VyOS networkAlso covers OSPF, BGP · Intermediate · 60 min

    Split one BGP router into edge routers that face the outside world and a core that carries traffic inside: loopbacks, OSPF for infrastructure, iBGP between loopbacks, next-hop-self, and what happens when a link fails. Built and broken on three real VyOS routers.