Failover between the ISPs
Two ISPs are only worth having if traffic moves when one fails. How quickly it moves depends on how the failure looks to each router.
A planned shutdown
To take ISP 1 down for maintenance, shut the session: neighbor 192.0.2.0 shutdown. FRR sends a NOTIFICATION, so both ends drop the session at once. edge switches its default route to ISP 2, and ISP 1 immediately uses its other path to the office, through ISP 2 (the prepended one). A ping from the server running throughout didn't lose a single packet.
An unplanned failure
Now the link to ISP 1 fails, but only on edge's side, as when a cable or an optic fails on one end. edge sees its interface go down, and switches to ISP 2 within the same second. ISP 1's interface stays up, though. It has no idea anything is wrong, keeps its session marked Established, and keeps sending the office's incoming traffic down the dead link. Only when no keepalive has arrived for its hold time (180 seconds with FRR's traditional defaults) does it give up, with Hold Timer Expired.
The server's ping shows the cost: outgoing packets reached the web server through ISP 2 at once, but the replies were lost for 156.8 seconds, because they came back through ISP 1.
That's the gap BFD closes: a fast hello protocol between the routers that notices a dead path in under a second and takes BGP down with it. BFD on VyOS measures it; FRR on Debian has the same bfdd daemon (turn it on in /etc/frr/daemons). Lower BGP timers (timers 10 30) are a simpler but slower alternative, and both ends must agree.
root@edge:~# vtysh -c "conf t" -c "router bgp 64500" -c "neighbor 192.0.2.0 shutdown"; sleep 3; vtysh -c "show bgp summary" | grep -A3 "^Neighbor"; ip route show default
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
192.0.2.0 4 64501 7 6 0 0 0 00:00:03 Idle (Admin) 0 isp1 (primary)
192.0.2.2 4 64502 9 5 5 0 0 00:01:21 2 1 isp2 (backup)
default nhid 18 via 192.0.2.2 dev eth1 proto bgp metric 20
root@isp1:~# vtysh -c "show bgp ipv4 203.0.113.0/24" | sed -n 1,6p
BGP routing table entry for 203.0.113.0/24, version 6
Paths: (1 available, best #1, table default)
Advertised to non peer-group peers:
198.51.100.2
64502 64500 64500 64500
198.51.100.2 from 198.51.100.2 (198.51.100.2)
root@server:~# tail -2 /tmp/p1.txt
75 packets transmitted, 75 received, 0% packet loss, time 15385ms
rtt min/avg/max/mdev = 0.059/0.175/0.324/0.066 ms
root@edge:~# vtysh -c "conf t" -c "router bgp 64500" -c "no neighbor 192.0.2.0 shutdown"- Lines 1–6: Shut the session: edge's default route moves to ISP 2 at once.
- Lines 8–14: ISP 1 now reaches the office through ISP 2, on the prepended path.
- Lines 16–18: Not one ping lost.
- Line 20: And back.
root@edge:~# date +%T; ip link set eth0 down; sleep 2; ip route show default
22:09:37
default nhid 18 via 192.0.2.2 dev eth1 proto bgp metric 20
root@isp1:~# vtysh -c "show bgp summary" | grep -A2 "^Neighbor"
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
192.0.2.1 4 64500 20 34 12 0 0 00:01:26 1 5 office (edge)
198.51.100.2 4 64502 23 22 12 0 0 00:07:37 1 4 N/A
root@isp1:~# vtysh -c "show bgp neighbors 192.0.2.1" | grep -E "BGP state|Last reset"
BGP state = Active
Last reset 00:01:01, Notification sent (Hold Timer Expired)
root@server:~# tail -2 /tmp/p2.txt
440 packets transmitted, 130 received, 70.4545% packet loss, time 221296ms
rtt min/avg/max/mdev = 0.075/0.174/0.956/0.154 ms
root@server:~# awk "/bytes from/{print \$1}" /tmp/p2.txt | tr -d "[]" | awk "NR>1 && \$1-p>1.5 {printf \"no replies for %.1f s\n\", \$1-p} {p=\$1}"
no replies for 156.8 s- Lines 1–3: edge's link to ISP 1 goes down; edge moves to ISP 2 within two seconds.
- Lines 5–8: ISP 1 still shows the session as up: its own interface is fine.
- Lines 10–12: Eventually, Hold Timer Expired.
- Lines 14–16: The server's ping over the whole test…
- Lines 18–19: …lost replies for 156.8 seconds.
Check your understanding
Why did the planned shutdown lose nothing, but the link failure lose replies for 157 seconds?
- A shutdown sends a NOTIFICATION, so ISP 1 learns at once; a one-sided link failure tells ISP 1 nothing until its hold timer expires.
- The link failure also broke ISP 2.
- edge was slow to switch after the link failure.
- Prepending only works during planned shutdowns.