Quiet failures, and the lab

When a cable is unplugged, both routers see the link go down and reroute in a moment. The failures that hurt are the quiet ones: the link stays up, but packets stop. A switch in the middle hangs, a provider's circuit fails somewhere you can't see, a fibre fails in one direction only. The interface says up, so nothing notices.

Nothing, that is, except the routing protocol's own keepalives, and they're slow on purpose:

Protocol	Hellos / keepalives	Declared dead after
OSPF (defaults)	every 10 s	40 s without one
BGP (FRR and VyOS defaults)	every 60 s	180 s without one

In Why not keep it simple? a quiet failure took far too long to notice. This lesson measures it on VyOS, then fixes it with Bidirectional Forwarding Detection (BFD): a tiny protocol whose only job is to answer "is the path to my neighbour still forwarding?", many times a second.

What the lab measured

	Without BFD	With BFD (300 ms × 3)
OSPF	about 35 s of lost traffic	under 1 s
eBGP	about 159 s	about 1 s

The lab

R1 is a VyOS VM, and R2 runs FRR (the same routing suite VyOS uses underneath). They're joined by two links: link A, the primary, and link B, the backup. Each has a LAN with one host, and h1 pings h2 ten times a second throughout, so every lost ping is a tenth of a second of outage.

To fail link A quietly, the lab unplugs it in the middle: inside R1's QEMU container, the wire is detached from the bridge that connects the VM to the Docker network. R1's interface and R2's interface both stay up, with carrier, exactly like a dead switch between them.

#!/bin/sh
# ./fail-test.sh OUTFILE SECS: ping h1 -> h2 ten times a second for SECS, quietly cut link A after 5 s,
# then put it back. Writes a transcript to out/OUTFILE.
cd "$(dirname "$0")"
f=out/$1; secs=$2; mkdir -p out
x() { docker compose exec -T "$@"; }
x h1 sh -c "setsid ping -i 0.1 -c $((secs * 10)) -q 198.51.100.130 > /tmp/ping.txt 2>&1 < /dev/null &"
sleep 5
printf 'root@r1-container:~# date +%%T; ip link set eth0 nomaster   # cut link A in the middle; carrier stays up\n' >> $f
x r1 sh -c 'date +%T; ip link set eth0 nomaster' >> $f
printf '\nroot@h1:~# ping -i 0.1 -c %s -q 198.51.100.130   # started 5 s before the cut\n' "$((secs * 10))" >> $f
sleep $((secs - 3))
x h1 sh -c 'until grep -q statistics /tmp/ping.txt; do sleep 1; done; sleep 1; cat /tmp/ping.txt' | grep -v '^$' >> $f
x r1 sh -c 'ip link set eth0 master br0 && bridge link set dev eth0 learning off'
echo >> $f