Try it yourself
The experiment takes about ten minutes to run yourself, most of it waiting for BGP's hold timer, and it runs on any machine with Docker, including an Apple Silicon Mac. Seeing the numbers on your own screen is more convincing than reading them here.
Put the files below in one directory, along with the four router configs from The experiment (r1-static.conf, r1-bgp.conf and their r2 twins, which differ only in addresses, AS number and names).
- Static design:
DESIGN=static docker compose -f lab-compose.yaml up -d --build. Ping r2's loopback from r1:docker compose -f lab-compose.yaml exec r1 ping -I 198.51.100.1 198.51.100.2. - Fail link A quietly on r2:
docker compose -f lab-compose.yaml exec r2 sh -c 'iptables -A INPUT -i eth0 -j DROP; iptables -A OUTPUT -o eth0 -j DROP', and watch the ping. - BGP design:
DESIGN=bgp docker compose -f lab-compose.yaml up -d --force-recreate, start the ping again, fail link A the same way, and time how long until replies return. - Add BFD in
vtyshon both routers (router bgp …→neighbor <link A and link B addresses> bfd), recreate, and repeat.
# Two routers, two links. Run with the static or BGP configs:
# DESIGN=static docker compose -f lab-compose.yaml up -d --build
# DESIGN=bgp docker compose -f lab-compose.yaml up -d --build
name: why-lab
x-router: &router
build: .
image: ttr-frr-why:10.2.1
privileged: true # iptables (to fail a link quietly) and ip link
services:
r1:
<<: *router
hostname: r1
volumes: ["./daemons:/etc/frr/daemons:ro", "./r1-${DESIGN:-static}.conf:/etc/frr/frr.conf:ro"]
networks: { linka: { interface_name: eth0 }, linkb: { interface_name: eth1 } }
r2:
<<: *router
hostname: r2
volumes: ["./daemons:/etc/frr/daemons:ro", "./r2-${DESIGN:-static}.conf:/etc/frr/frr.conf:ro"]
networks: { linka: { interface_name: eth0 }, linkb: { interface_name: eth1 } }
networks:
linka: { enable_ipv4: false, enable_ipv6: true, ipam: { config: [{ subnet: "fd00:aa::/64" }] } }
linkb: { enable_ipv4: false, enable_ipv6: true, ipam: { config: [{ subnet: "fd00:bb::/64" }] } }# FRR 10.2.1 plus the tools to fail a link quietly and measure the outage.
FROM quay.io/frrouting/frr:10.2.1
RUN apk add --no-cache iptables iputils-ping \
&& mkdir -p /var/log/frr && chown frr:frr /var/log/frr \
&& echo "service integrated-vtysh-config" > /etc/frr/vtysh.confbgpd=yes
ospfd=no
ospf6d=no
ripd=no
ripngd=no
isisd=no
pimd=no
ldpd=no
nhrpd=no
eigrpd=no
babeld=no
sharpd=no
pbrd=no
bfdd=yes
fabricd=no
vrrpd=no
pathd=no
staticd=yes
vtysh_enable=yes
zebra_options=" -A 127.0.0.1 -s 90000000"
bgpd_options=" -A 127.0.0.1"
staticd_options="-A 127.0.0.1"
bfdd_options=" -A 127.0.0.1"frr version 10.2.1
frr defaults traditional
hostname r2
log file /var/log/frr/frr.log informational
service integrated-vtysh-config
!
interface eth0
description link A
ip address 192.0.2.1/31
exit
!
interface eth1
description link B
ip address 192.0.2.3/31
exit
!
interface lo
ip address 198.51.100.2/32
exit
!
! The "simple" design: reach r1 over link A, with a floating backup over link B
! (administrative distance 200, so it's only used if the first route disappears).
ip route 198.51.100.1/32 192.0.2.0
ip route 198.51.100.1/32 192.0.2.2 200frr version 10.2.1
frr defaults traditional
hostname r2
log file /var/log/frr/frr.log informational
service integrated-vtysh-config
!
interface eth0
description link A
ip address 192.0.2.1/31
exit
!
interface eth1
description link B
ip address 192.0.2.3/31
exit
!
interface lo
ip address 198.51.100.2/32
exit
!
! The routing-protocol design: a BGP session over each link, each announcing
! r2's loopback. Whichever session is alive carries the traffic.
router bgp 64501
bgp router-id 198.51.100.2
bgp log-neighbor-changes
neighbor 192.0.2.0 remote-as 64500
neighbor 192.0.2.0 description r1 via link A
neighbor 192.0.2.2 remote-as 64500
neighbor 192.0.2.2 description r1 via link B
!
address-family ipv4 unicast
network 198.51.100.2/32
neighbor 192.0.2.0 route-map ALLOW in
neighbor 192.0.2.0 route-map ALLOW out
neighbor 192.0.2.2 route-map ALLOW in
neighbor 192.0.2.2 route-map ALLOW out
maximum-paths 1
exit-address-family
exit
!
! A two-router lab: accept and send everything. Real edges filter (see the BGP lessons).
route-map ALLOW permit 10
exitWhere next
- BGP on loopbacks vs interface addressesAlso covers BGP, BFD · Intermediate · 45 min
Why BGP sessions between loopbacks survive link failures and use parallel links, and what they need to work: an IGP or static routes, update-source, the eBGP connected check, and BFD. Every failure is reproduced on two real VyOS routers.
- BGP FundamentalsAlso covers BGP · Intermediate · 75 min
How the Border Gateway Protocol glues the internet together: autonomous systems, sessions, path attributes, best-path selection, iBGP scaling and a hands-on eBGP lab with real FRR configuration.
- Route reflectors on VyOSAlso covers BGP · Intermediate · 30 min
Replace an iBGP full mesh with route reflectors: see iBGP refuse to pass routes on, turn on reflection, read ORIGINATOR_ID and CLUSTER_LIST, confirm the reflector stays out of the forwarding path, and survive taking one reflector down.
- IPv4 over an IPv6-only core (RFC 8950)Also covers BGP · Intermediate · 40 min
Carry IPv4 routes over IPv6 BGP sessions with IPv6 next hops, so core routers need no IPv4 addresses at all. Built on VyOS: the missing router ID, what breaks without the extended next hop capability, native IPv4 on the wire, and why traceroute shows 192.0.0.8.