TTR Networks

Beginner · Why not just keep it simple? · section 3 of 8

The experiment

✓ Lab-tested 18 October 2026 on FRR 10.2.1

Arguments about network design are easy to have and hard to settle, so this one is settled with an experiment. Two routers are joined by two links, and the same failure is thrown at three designs: static routes with a backup, BGP with its default settings, and BGP with fast failure detection (BFD).

The failure is the quiet kind: link A stops carrying traffic, but every interface stays up, as when something in the middle of a provider's link dies. In the lab, a firewall rule on r2 drops everything on link A, which looks identical from r1. Throughout, r1 pings r2's loopback, and the ping statistics measure the outage.

DesignWhat happenedOutage
Static route, with a floating backup over link BNothing. r1 kept sending into link A.Until someone fixes it (100% loss)
…and even if r1's own interface goes downr1 moved to link B, but r2 kept replying over link AUntil someone fixes it (100% loss)
BGP over both links, default timersBoth routers dropped the link A session when its hold timer expired, and moved to link BAbout 170 seconds
BGP over both links, with BFDBFD declared link A dead almost at once; BGP moved to link BAbout 0.7 seconds

Three things are worth taking away:

  1. The "simple" backup didn't back anything up. A floating static route only covers the one failure that takes an interface down, and even then only at one end. It looks redundant on a diagram, which is worse than having no backup at all, because nobody plans for its failure.
  2. A routing protocol made the failure automatic, at both ends at once, because it noticed the silence rather than the interface lights.
  3. Defaults matter. BGP's default hold time (180 seconds in FRR's traditional profile) is a sensible protection against flapping across the internet, and a long outage on a link you own. BFD turned three minutes into under a second. That's a small amount of extra configuration, and you only know to add it once you've measured.
Diagram: Two routers, two links, one quiet failure — Step through what each design does when link A fails quietly.
  1. Normal: traffic uses link A. Both designs, static and BGP, send traffic between the loopbacks over link A, with link B as the backup.
  2. The quiet failure. Something in the middle of link A stops forwarding. Both routers' interfaces stay up, so neither router's lights say anything is wrong.
  3. Static routing: sending into the void. r1's static route still points at link A, because its interface is up. The floating backup over link B never activates. 100% loss, until a person intervenes.
  4. Even half a fix doesn't help. If r1's own interface does go down, r1 moves to link B. But r2 can't see the failure and still replies over link A: still 100% loss.
  5. A routing protocol: both ends notice. BGP keepalives stop arriving over link A, so both routers drop that session and move to link B together. Default timers take minutes; BFD does it in under a second.
r1: the static design
frr version 10.2.1
frr defaults traditional
hostname r1
log file /var/log/frr/frr.log informational
service integrated-vtysh-config
!
interface eth0
 description link A
 ip address 192.0.2.0/31
exit
!
interface eth1
 description link B
 ip address 192.0.2.2/31
exit
!
interface lo
 ip address 198.51.100.1/32
exit
!
! The "simple" design: reach r2 over link A, with a floating backup over link B
! (administrative distance 200, so it's only used if the first route disappears).
ip route 198.51.100.2/32 192.0.2.1
ip route 198.51.100.2/32 192.0.2.3 200
  • Line 23: The main route to r2's loopback, over link A.
  • Line 24: The floating backup over link B. Distance 200 means it's only installed if the main route is removed.
Static routing, before and after link A fails quietly (real output)
r1# show ip route 198.51.100.2
Routing entry for 198.51.100.2/32
  Known via "static", distance 200, metric 0
  Last update 00:00:09 ago
    192.0.2.3, via eth1, weight 1

Routing entry for 198.51.100.2/32
  Known via "static", distance 1, metric 0, best
  Last update 00:00:09 ago
  * 192.0.2.1, via eth0, weight 1


r1:~# ping -q -c 10 -i 0.2 -I 198.51.100.1 198.51.100.2
PING 198.51.100.2 (198.51.100.2) from 198.51.100.1 : 56(84) bytes of data.

--- 198.51.100.2 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 1866ms
rtt min/avg/max/mdev = 0.039/0.052/0.085/0.012 ms

r2:~# date "+%H:%M:%S link A now carries nothing (its interfaces stay up)"
04:54:11 link A now carries nothing (its interfaces stay up)

r1:~# ip -br link show eth0
eth0@if1481      UP             c2:82:97:73:df:fe <BROADCAST,MULTICAST,UP,LOWER_UP> 

r1# show ip route 198.51.100.2
Routing entry for 198.51.100.2/32
  Known via "static", distance 200, metric 0
  Last update 00:00:11 ago
    192.0.2.3, via eth1, weight 1

Routing entry for 198.51.100.2/32
  Known via "static", distance 1, metric 0, best
  Last update 00:00:11 ago
  * 192.0.2.1, via eth0, weight 1


r1:~# ping -q -c 50 -i 0.2 -I 198.51.100.1 198.51.100.2
PING 198.51.100.2 (198.51.100.2) from 198.51.100.1 : 56(84) bytes of data.

--- 198.51.100.2 ping statistics ---
50 packets transmitted, 0 received, 100% packet loss, time 10190ms
  • Lines 1–10: Both routes are configured; only the link A one (distance 1) is in use.
  • Lines 13–18: Healthy: every ping answered.
  • Lines 20–21: In the lab, a firewall rule on r2 makes link A carry nothing, playing the part of a fault in the middle.
  • Lines 23–24: r1's link A interface is still UP. As far as r1 can tell, nothing has happened…
  • Lines 26–35: …so the routing table is unchanged, still pointing into the dead link.
  • Lines 38–42: 100% loss, indefinitely. The backup route sits unused.
Static routing when r1's own interface does go down (real output)
r1:~# ip link set eth0 down

r1# show ip route 198.51.100.2
Routing entry for 198.51.100.2/32
  Known via "static", distance 200, metric 0, best
  Last update 00:00:32 ago
  * 192.0.2.3, via eth1, weight 1


r1:~# ping -q -c 10 -i 0.2 -I 198.51.100.1 198.51.100.2
PING 198.51.100.2 (198.51.100.2) from 198.51.100.1 : 56(84) bytes of data.

--- 198.51.100.2 ping statistics ---
10 packets transmitted, 0 received, 100% packet loss, time 1867ms
  • Lines 3–7: This time r1 notices: its interface is down, so the link A route is removed and the floating backup takes over.
  • Lines 10–14: Still 100% loss. r2's link A interface is still up, so r2 still sends its replies into link A. Static routes only know about their own end.
r1: the BGP design
frr version 10.2.1
frr defaults traditional
hostname r1
log file /var/log/frr/frr.log informational
service integrated-vtysh-config
!
interface eth0
 description link A
 ip address 192.0.2.0/31
exit
!
interface eth1
 description link B
 ip address 192.0.2.2/31
exit
!
interface lo
 ip address 198.51.100.1/32
exit
!
! The routing-protocol design: a BGP session over each link, each announcing
! r1's loopback. Whichever session is alive carries the traffic.
router bgp 64500
 bgp router-id 198.51.100.1
 bgp log-neighbor-changes
 neighbor 192.0.2.1 remote-as 64501
 neighbor 192.0.2.1 description r2 via link A
 neighbor 192.0.2.3 remote-as 64501
 neighbor 192.0.2.3 description r2 via link B
 !
 address-family ipv4 unicast
  network 198.51.100.1/32
  neighbor 192.0.2.1 route-map ALLOW in
  neighbor 192.0.2.1 route-map ALLOW out
  neighbor 192.0.2.3 route-map ALLOW in
  neighbor 192.0.2.3 route-map ALLOW out
  maximum-paths 1
 exit-address-family
exit
!
! A two-router lab: accept and send everything. Real edges filter (see the BGP lessons).
route-map ALLOW permit 10
exit
  • Lines 26–29: One eBGP session over each link.
  • Line 32: Announce r1's loopback over both.
  • Line 37: Use one path at a time, so the test shows failover, not load sharing.
BGP with default timers, same quiet failure (real output)
r1# show bgp summary

IPv4 Unicast Summary:
BGP router identifier 198.51.100.1, local AS number 64500 VRF default vrf-id 0
BGP table version 2
RIB entries 3, using 384 bytes of memory
Peers 2, using 33 KiB of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
192.0.2.1       4      64501        12        14        2    0    0 00:00:11            1        2 r2 via link A
192.0.2.3       4      64501        12        14        2    0    0 00:00:11            1        2 r2 via link B

Total number of neighbors 2

r1# show ip route 198.51.100.2
Routing entry for 198.51.100.2/32
  Known via "bgp", distance 20, metric 0, best
  Last update 00:00:10 ago
  * 192.0.2.1, via eth0, weight 1


r2:~# date "+%H:%M:%S link A now carries nothing (its interfaces stay up)"
04:55:17 link A now carries nothing (its interfaces stay up)

r1:~# ping -q -w 240 -i 0.5 -I 198.51.100.1 198.51.100.2
PING 198.51.100.2 (198.51.100.2) from 198.51.100.1 : 56(84) bytes of data.

--- 198.51.100.2 ping statistics ---
476 packets transmitted, 137 received, 71.2185% packet loss, time 239800ms
rtt min/avg/max/mdev = 0.029/0.178/1.283/0.112 ms

r1:~# grep -E "ADJCHANGE|Hold Timer" /var/log/frr/frr.log | tail -3
2026/10/18 04:55:06 BGP: [N9HHH-F8H1M] %ADJCHANGE: neighbor 192.0.2.3(r2) in vrf default Up
2026/10/18 04:58:08 BGP: [HZN6M-XRM1G] %NOTIFICATION: sent to neighbor 192.0.2.1 4/0 (Hold Timer Expired) 0 bytes 
2026/10/18 04:58:08 BGP: [PXVXG-TFNNT] %ADJCHANGE: neighbor 192.0.2.1(r2) in vrf default Down BGP Notification send

r1# show ip route 198.51.100.2
Routing entry for 198.51.100.2/32
  Known via "bgp", distance 20, metric 0, best
  Last update 00:01:10 ago
  * 192.0.2.3, via eth1, weight 1


r1:~# ping -q -c 10 -i 0.2 -I 198.51.100.1 198.51.100.2
PING 198.51.100.2 (198.51.100.2) from 198.51.100.1 : 56(84) bytes of data.

--- 198.51.100.2 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 1868ms
rtt min/avg/max/mdev = 0.042/0.130/0.156/0.031 ms
  • Lines 10–11: Two sessions, one per link, both established.
  • Line 19: Traffic uses link A.
  • Lines 22–23: Link A goes quiet at 04:55:17.
  • Line 29: Over 240 seconds of pings at half-second intervals, 339 were lost: about 170 seconds of outage.
  • Lines 34–35: 04:58:08: no keepalive for 180 seconds, so the link A session is declared dead. Both routers move to link B.
  • Line 41: Now via link B…
  • Line 48: …and working, with nobody touching anything.
BGP with BFD, same quiet failure (real output)
r1# show bfd peers brief
Session count: 2
SessionId  LocalAddress                             PeerAddress                             Status         
=========  ============                             ===========                             ======         
3253519194 192.0.2.0                                192.0.2.1                               up             
4285188274 192.0.2.2                                192.0.2.3                               up             

r2:~# date "+%H:%M:%S link A now carries nothing (its interfaces stay up)"
04:59:43 link A now carries nothing (its interfaces stay up)

r1:~# ping -q -w 20 -i 0.1 -I 198.51.100.1 198.51.100.2
PING 198.51.100.2 (198.51.100.2) from 198.51.100.1 : 56(84) bytes of data.

--- 198.51.100.2 ping statistics ---
191 packets transmitted, 184 received, 3.66492% packet loss, time 19900ms
rtt min/avg/max/mdev = 0.024/0.088/0.493/0.054 ms

r1:~# grep -E "ADJCHANGE|BFD|bfd" /var/log/frr/frr.log | tail -3
2026/10/18 04:59:24 BGP: [N9HHH-F8H1M] %ADJCHANGE: neighbor 192.0.2.3(r2) in vrf default Up
2026/10/18 04:59:44 BGP: [HZN6M-XRM1G] %NOTIFICATION(Hard Reset): sent to neighbor 192.0.2.1 6/10 (Cease/BFD Down) 0 bytes 
2026/10/18 04:59:44 BGP: [PXVXG-TFNNT] %ADJCHANGE: neighbor 192.0.2.1(r2) in vrf default Down BGP Notification send

r1# show bfd peers brief
Session count: 2
SessionId  LocalAddress                             PeerAddress                             Status         
=========  ============                             ===========                             ======         
3253519194 192.0.2.0                                192.0.2.1                               down           
4285188274 192.0.2.2                                192.0.2.3                               up             

r1# show ip route 198.51.100.2
Routing entry for 198.51.100.2/32
  Known via "bgp", distance 20, metric 0, best
  Last update 00:00:39 ago
  * 192.0.2.3, via eth1, weight 1
  • Lines 5–6: A BFD session on each link, alongside BGP (neighbor … bfd on both routers).
  • Lines 8–9: Link A goes quiet at 04:59:43.
  • Line 15: 7 pings lost at 100 ms intervals: about 0.7 seconds of outage.
  • Lines 20–21: 04:59:44: BFD declares link A down and BGP drops that session immediately.
  • Lines 27–28: BFD sees exactly which link is dead.
  • Line 34: Traffic on link B within a second.