TTR Networks

Intermediate · Debugging BGP: every session state, reproduced · section 2 of 8

The lab

✓ Lab-tested 18 October 2026 on FRR 10.2.1, Docker Desktop (Linux kernel without TCP-MD5)

The lab is three FRR routers in Docker: r1 in AS 64500 runs eBGP to r2 in AS 64501, and r2 runs iBGP to r3. It starts healthy, and each scenario breaks exactly one thing, then puts it back.

r1 originates 203.0.113.0/24. r2 originates three /26s from 198.51.100.0/24, and passes on a /27 it learns from r3. So a healthy r1 receives 4 prefixes and sends 1, and those two numbers are the quickest health check there is.

Each router's container also has iptables (to play the part of a firewall) and tcpdump. Building the lab yourself is covered in the last section.

Diagram: The BGP debugging lab — Step through to see which session each scenario breaks.
  1. The session we break. Almost every scenario breaks the eBGP session between r1 (AS 64500) and r2 (AS 64501), on a directly connected /31.
  2. What a healthy r1 receives. r2 announces its three /26s and passes on r3's /27, so r1 should receive exactly 4 prefixes and send 1: 203.0.113.0/24.
  3. iBGP inside AS 64501. r2 and r3 peer over iBGP. r2 sets next-hop-self towards r3; one scenario removes that and r3 can no longer use r1's route.
  4. A session two hops long. One scenario adds eBGP between r1's and r3's loopbacks, routed through r2. eBGP sends with TTL 1 by default, so r2 can't forward r1's SYN and answers with ICMP time exceeded.
  5. With ebgp-multihop 2. Now r1 sends with TTL 2: r2 forwards the SYN with one hop to spare, and r3 receives it.
r1 — /etc/frr/frr.conf
frr version 10.2.1
frr defaults traditional
hostname r1
log file /var/log/frr/frr.log informational
service integrated-vtysh-config
!
interface eth0
 ip address 192.0.2.0/31
exit
!
interface lo
 ip address 198.51.100.241/32
exit
!
ip route 203.0.113.0/24 blackhole
ip route 198.51.100.243/32 192.0.2.1
!
router bgp 64500
 bgp router-id 198.51.100.241
 bgp log-neighbor-changes
 neighbor 192.0.2.1 remote-as 64501
 neighbor 192.0.2.1 description r2
 !
 address-family ipv4 unicast
  network 203.0.113.0/24
  neighbor 192.0.2.1 soft-reconfiguration inbound
  neighbor 192.0.2.1 route-map FROM-R2 in
  neighbor 192.0.2.1 route-map TO-R2 out
 exit-address-family
exit
!
ip prefix-list OURS seq 10 permit 203.0.113.0/24
ip prefix-list FROM-R2 seq 10 permit 198.51.100.0/24 le 27
!
route-map TO-R2 permit 10
 match ip address prefix-list OURS
exit
!
route-map FROM-R2 permit 10
 match ip address prefix-list FROM-R2
exit
  • Line 4: Log to a file, so you can read and search what happened after the fact.
  • Lines 15–16: r1's own prefix (a blackhole route so network has something to match), and the route to r3's loopback used by the multihop scenario.
  • Line 20: Log every session up/down with its reason. Not on by default in FRR's traditional profile.
  • Lines 21–22: The session that most scenarios break: eBGP to r2's interface address.
  • Line 26: Keeps a copy of everything r2 sends before policy, so received-routes works. Essential for the import-filter scenario.
  • Lines 27–28: Import and export policy. Without these, FRR refuses to exchange any routes on an eBGP session.
  • Line 33: Accept anything inside 198.51.100.0/24 up to /27: r2's three /26s and r3's /27.
r2 — /etc/frr/frr.conf
frr version 10.2.1
frr defaults traditional
hostname r2
log file /var/log/frr/frr.log informational
service integrated-vtysh-config
!
interface eth0
 ip address 192.0.2.1/31
exit
!
interface eth1
 ip address 192.0.2.2/31
exit
!
interface lo
 ip address 198.51.100.242/32
exit
!
ip route 198.51.100.0/26 blackhole
ip route 198.51.100.64/26 blackhole
ip route 198.51.100.128/26 blackhole
ip route 198.51.100.241/32 192.0.2.0
ip route 198.51.100.243/32 192.0.2.3
!
router bgp 64501
 bgp router-id 198.51.100.242
 bgp log-neighbor-changes
 neighbor 192.0.2.0 remote-as 64500
 neighbor 192.0.2.0 description r1
 neighbor 192.0.2.3 remote-as 64501
 neighbor 192.0.2.3 description r3 (iBGP)
 !
 address-family ipv4 unicast
  network 198.51.100.0/26
  network 198.51.100.64/26
  network 198.51.100.128/26
  neighbor 192.0.2.0 route-map FROM-R1 in
  neighbor 192.0.2.0 route-map TO-R1 out
  neighbor 192.0.2.3 next-hop-self
 exit-address-family
exit
!
ip prefix-list R1 seq 10 permit 203.0.113.0/24
ip prefix-list OURS seq 10 permit 198.51.100.0/24 le 27
!
route-map FROM-R1 permit 10
 match ip address prefix-list R1
exit
!
route-map TO-R1 permit 10
 match ip address prefix-list OURS
exit
  • Lines 28–31: An eBGP session to r1 and an iBGP session to r3 (same AS, 64501).
  • Lines 34–36: The three prefixes r2 originates towards r1.
  • Line 39: r3 learns r1's routes with r2 as the next hop. One scenario removes this line.
r3 — /etc/frr/frr.conf
frr version 10.2.1
frr defaults traditional
hostname r3
log file /var/log/frr/frr.log informational
service integrated-vtysh-config
!
interface eth0
 ip address 192.0.2.3/31
exit
!
interface lo
 ip address 198.51.100.243/32
exit
!
ip route 198.51.100.192/27 blackhole
ip route 198.51.100.241/32 192.0.2.2
!
router bgp 64501
 bgp router-id 198.51.100.243
 bgp log-neighbor-changes
 neighbor 192.0.2.2 remote-as 64501
 neighbor 192.0.2.2 description r2 (iBGP)
 !
 address-family ipv4 unicast
  network 198.51.100.192/27
 exit-address-family
exit
  • Line 14: r3's own /27, which r2 passes on to r1.
  • Line 15: A route back to r1's loopback for the multihop scenario. Note there's no route to r1's link (192.0.2.0/31).
r1: the healthy baseline (real output)
r1# show bgp summary

IPv4 Unicast Summary:
BGP router identifier 198.51.100.241, local AS number 64500 VRF default vrf-id 0
BGP table version 5
RIB entries 9, using 1152 bytes of memory
Peers 1, using 17 KiB of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
192.0.2.1       4      64501        10         8        5    0    0 00:00:21            4        1 r2

Total number of neighbors 1

r1# show bgp ipv4 unicast
BGP table version is 5, local router ID is 198.51.100.241, vrf id 0
Default local pref 100, local AS 64500
Status codes:  s suppressed, d damped, h history, u unsorted, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>  198.51.100.0/26  192.0.2.1                0             0 64501 i
 *>  198.51.100.64/26 192.0.2.1                0             0 64501 i
 *>  198.51.100.128/26
                    192.0.2.1                0             0 64501 i
 *>  198.51.100.192/27
                    192.0.2.1                              0 64501 i
 *>  203.0.113.0/24   0.0.0.0                  0         32768 i

Displayed 5 routes and 5 total paths
  • Line 10: The two numbers to remember: 4 prefixes received, 1 sent. Anything else means something is wrong.
  • Lines 24–30: r2's three /26s, r3's /27 (via r2), and r1's own /24 (next hop 0.0.0.0 means "originated here").

Every state, in one log

Before breaking anything, watch the session come up once with FRR's state-machine debugging turned on. In a healthy session, OpenSent and OpenConfirm last a few milliseconds, so this is the only way you'll see them.

The log has a twist that confuses people the first time: there are two connections. r1 dialled r2 (file descriptor 22), and at the same moment r2 dialled r1 (file descriptor 26). Two TCP connections for one session is a collision. BGP resolves it by keeping the connection opened by the router with the higher router ID, and closing the other with a NOTIFICATION. r2's ID (198.51.100.242) is higher, so r2's connection survives. Seeing both paths at once is a bonus: fd 22 goes through Connect, and fd 26 through Active.

r1: one session coming up, through every state (real output)
r1:~# grep -E "went from|ADJCHANGE|NOTIFICATION|sending OPEN|rcv OPEN," /var/log/frr/frr.log
2026/10/18 03:26:29 BGP: [H7QV4-WR3ZD] %NOTIFICATION(Hard Reset): sent to neighbor 192.0.2.1 6/4 (Cease/Administrative Reset) ""
2026/10/18 03:26:29 BGP: [PXVXG-TFNNT] %ADJCHANGE: neighbor 192.0.2.1(r2) in vrf default Down User reset
2026/10/18 03:26:29 BGP: [HKWM3-ZC5QP] 192.0.2.1 fd -1 went from Established to Clearing
2026/10/18 03:26:29 BGP: [HKWM3-ZC5QP] 192.0.2.1 fd -1 went from Clearing to Idle
2026/10/18 03:26:31 BGP: [HKWM3-ZC5QP] 192.0.2.1 fd 22 went from Idle to Connect
2026/10/18 03:26:31 BGP: [HKWM3-ZC5QP] 192.0.2.1 fd 26 went from Idle to Active
2026/10/18 03:26:31 BGP: [G1CME-TC11B] 192.0.2.1(Unknown) fd 26 sending OPEN, version 4, my as 64500, holdtime 180, id 198.51.100.241
2026/10/18 03:26:31 BGP: [HKWM3-ZC5QP] 192.0.2.1 fd 26 went from Active to OpenSent
2026/10/18 03:26:31 BGP: [G1CME-TC11B] 192.0.2.1(r2) fd 22 sending OPEN, version 4, my as 64500, holdtime 180, id 198.51.100.241
2026/10/18 03:26:31 BGP: [HKWM3-ZC5QP] 192.0.2.1 fd 22 went from Connect to OpenSent
2026/10/18 03:26:31 BGP: [WNM1E-D314G] 192.0.2.1 rcv OPEN, version 4, remote-as (in open) 64501, holdtime 180, id 198.51.100.242
2026/10/18 03:26:31 BGP: [HZN6M-XRM1G] %NOTIFICATION: sent to neighbor 192.0.2.1 6/7 (Cease/Connection Collision Resolution) 0 bytes 
2026/10/18 03:26:31 BGP: [HKWM3-ZC5QP] 192.0.2.1 fd 26 went from OpenSent to OpenConfirm
2026/10/18 03:26:31 BGP: [HKWM3-ZC5QP] 192.0.2.1 fd -1 went from OpenSent to Idle
2026/10/18 03:26:31 BGP: [HKWM3-ZC5QP] 192.0.2.1 fd 26 went from OpenConfirm to Established
2026/10/18 03:26:31 BGP: [N9HHH-F8H1M] %ADJCHANGE: neighbor 192.0.2.1(r2) in vrf default Up
2026/10/18 03:26:31 BGP: [HKWM3-ZC5QP] 192.0.2.1 fd -1 went from Idle to Deleted
  • Line 1: debug bgp neighbor-events plus a debugging-level log file, then clear bgp 192.0.2.1. The grep keeps just the state changes and OPENs.
  • Lines 2–5: The clear: r1 tells r2 it's resetting (Cease/Administrative Reset), then the session drops through Clearing to Idle.
  • Line 6: fd 22: the connection r1 dials out. Idle → Connect means a SYN has been sent.
  • Line 7: fd 26: at the same moment r2 dials r1. r1 accepts it, and that connection starts in Active.
  • Lines 8–11: Both connections reach OpenSent: r1 has sent an OPEN on each.
  • Line 12: r2's OPEN arrives: AS 64501, hold time 180 s, router ID 198.51.100.242.
  • Lines 13–15: Two connections for one session is a collision. BGP keeps the one opened by the router with the higher router ID (r2's, fd 26) and closes the other with Cease/Connection Collision Resolution.
  • Lines 14–16: The survivor goes OpenSent → OpenConfirm (OPENs agreed) → Established (first KEEPALIVE received).
  • Line 17: What log-neighbor-changes writes on its own, without any debugging: the session is Up.

Check your understanding

r1's summary shows r2 established with 3 prefixes received instead of the usual 4. Which state is the session in, and where do you look first?

  • Established; this is a route problem, so look at what r2 sent and what r1's import policy kept.
  • OpenConfirm; one OPEN capability didn't match.
  • Active; r1 is retrying the connection.
  • Idle (PfxCt); the max-prefix limit is 3.