When a link fails
This is what the design is for. Disable core1's link to edge2 (set interfaces ethernet eth1 disable on core1, then commit) and watch what changes.
- OSPF reacts. The adjacency with edge2 goes away, and core1's route to edge2's loopback moves to the path via edge1 (metric 2 instead of 1).
- BGP doesn't notice. The iBGP session to 198.51.100.243 is between loopbacks, and the loopback is still reachable, so the session stays up. No routes are withdrawn, nothing is re-advertised to the upstream or the IX, and nobody outside your AS sees anything happen.
- Traffic reroutes. The member's prefix still has next hop 198.51.100.243, but that next hop now resolves via edge1. Traffic to the IX goes core1 → edge1 → edge2.
If the iBGP sessions were between link addresses instead, losing the link would have dropped the session, withdrawn the IX routes from core1, and sent IX traffic out through transit until the link came back.
How fast does the reroute happen? core1 disabled its own interface, so it reacted at once. edge2's interface stayed up, so edge2 only noticed when the OSPF dead interval expired: 40 seconds by default. The same applies to any failure a router can't see directly, such as a fibre cut beyond a media converter or a switch in the middle. Production networks add Bidirectional Forwarding Detection (BFD) to cut that to well under a second.
Re-enable the link afterwards with delete interfaces ethernet eth1 disable.
- OSPF learns the loopbacks. OSPF runs only on the three internal links and the loopbacks. Within seconds every router can reach every other router's loopback, by more than one path.
- iBGP between loopbacks. Each router has an iBGP session to both of the others, sourced from and addressed to loopbacks. core1 originates our /24 aggregate and the LAN /25 into iBGP.
- The edges talk to the outside. edge1 announces only the /24 to the upstream and receives a default route. edge2 announces only the /24 to the route server and receives the member's prefix. All external filtering lives here.
- Next-hop-self. The edges pass external routes into iBGP with themselves as the next hop. Otherwise core1 would see next hops on links it has no route to, and couldn't use the routes at all.
- Traffic to the IX member. core1 prefers the IX path (local preference 150, set by edge2's import policy and carried across iBGP), so it forwards to edge2's loopback, which OSPF resolves to the direct link.
- Everything else via edge1. The default route points at edge1's loopback, so all other traffic leaves through the upstream.
- A link fails. The core1–edge2 link goes down. OSPF reroutes edge2's loopback via edge1, the iBGP session to edge2 stays up, and IX traffic now takes core1 → edge1 → edge2. No BGP routes change.
vyos@core1:~$ show ip ospf neighbor
Neighbor ID Pri State Up Time Dead Time Address Interface RXmtL RqstL DBsmL
198.51.100.242 1 Full/- 1m42s 35.767s 198.51.100.224 eth0:198.51.100.225 1 0 0
vyos@core1:~$ show ip route 198.51.100.243
Routing entry for 198.51.100.243/32
Known via "ospf", distance 110, metric 2, best
Last update 00:00:06 ago
Flags: Selected
Status: Installed
* 198.51.100.224, via eth0, weight 1
vyos@core1:~$ show ip bgp summary
IPv4 Unicast Summary:
BGP router identifier 198.51.100.241, local AS number 64500 VRF default vrf-id 0
BGP table version 8
RIB entries 5, using 800 bytes of memory
Peers 2, using 60 KiB of memory
Peer groups 1, using 72 bytes of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
198.51.100.242 4 64500 6 5 8 0 0 00:01:26 2 2 edge1
198.51.100.243 4 64500 6 5 8 0 0 00:01:26 1 2 edge2
Total number of neighbors 2
vyos@core1:~$ show ip route 203.0.113.64/26
Routing entry for 203.0.113.64/26
Known via "bgp", distance 200, metric 0, best
Last update 00:00:08 ago
Flags: Recursion iBGP Selected
Status: Installed
198.51.100.243 (recursive), weight 1
* 198.51.100.224, via eth0, weight 1- Line 4: Only edge1 is left as an OSPF neighbor.
- Lines 8–12: edge2's loopback is now two hops away, via edge1.
- Lines 24–25: Both iBGP sessions still up, with the same uptime as before the failure and the same prefix counts.
- Lines 34–35: Same BGP next hop (edge2's loopback), now resolved via eth0 to edge1.
root@host:~# traceroute -n 203.0.113.65
traceroute to 203.0.113.65 (203.0.113.65), 30 hops max, 60 byte packets
1 198.51.100.1 6.871 ms 7.184 ms 7.165 ms
2 198.51.100.224 8.502 ms 8.479 ms 8.469 ms
3 198.51.100.229 9.381 ms 9.376 ms 9.369 ms
4 203.0.113.65 9.460 ms 9.936 ms 9.935 ms- Line 4: edge1…
- Lines 5–6: …then edge2 over the edge-to-edge link, then the member.
Check your understanding
The iBGP sessions were configured between link addresses instead of loopbacks. What happens when the core1–edge2 link fails?
- Nothing different; OSPF reroutes as before
- The core1–edge2 session drops: core1 loses the IX routes and edge2 loses our aggregate, so IX traffic goes via transit in both directions
- The upstream session drops
- The OSPF adjacency between edge1 and edge2 drops