TTR Networks

Intermediate · Debugging BGP: every session state, reproduced · section 5 of 8

OpenSent and the OPEN message

✓ Lab-tested 18 October 2026 on FRR 10.2.1, Docker Desktop (Linux kernel without TCP-MD5)

When TCP connects but the session still won't come up, the routers have reached each other and disagreed about something in the OPEN message, and they say what in a NOTIFICATION. These failures are the easiest to diagnose, because the reason is written in the log in plain words.

The wrong remote AS

r1 is configured with remote-as 64999, but r2 is in AS 64501. TCP connects, r2 sends its OPEN saying "my AS is 64501", and r1 rejects it with NOTIFICATION 2/2: OPEN Message Error / Bad Peer AS. The session goes back to Idle, waits a moment, and tries again, over and over.

Both ends log it: r1 says it sent the NOTIFICATION, and r2 says it received it. That tells you which side's configuration disagrees with reality. The same pattern, with a different subcode, covers a bad router ID (2/3) and unacceptable hold times (2/6).

eBGP between loopbacks: TTL and the connected check

Here r1 and r3 try to peer between their loopbacks, two hops apart, with r2 routing between them. The loopbacks can ping each other, yet the session stays in Active, and the neighbour detail hints at why: External BGP neighbor may be up to 1 hops away.

eBGP assumes the peer is directly connected, so it sends its packets with a time-to-live (TTL) of 1. The capture on r2 shows exactly that: r1's SYN arrives with ttl 1, so r2 can't forward it, and replies with ICMP time exceeded instead.

ebgp-multihop 2 on both ends raises the TTL and allows a non-connected peer. Then clear the session. In this lab the first clear, straight after the change, wasn't enough, and a second one a few seconds later brought the session up: if a fix doesn't seem to work, clear again before you decide it's wrong. It arrives in a state you'll meet again later: (Policy), because this new eBGP session has no import or export policy yet. For directly connected routers peering between loopbacks, disable-connected-check is the narrower fix.

A password that isn't there

TCP-MD5 authentication (neighbor … password) signs every BGP segment. On a real router, a password configured on only one side, or a different one on each side, makes the kernel silently drop the other side's segments. The session sits in Connect or Active with nothing in the BGP log, just like the firewall case. The clue is in the kernel: its MD5 failure counters climb (nstat -az | grep -i md5), and many kernels also log the mismatch.

This lab can't reproduce that, and the reason is worth knowing. The kernel underneath Docker Desktop is built without TCP-MD5 support. When r1 is given a password and r2 isn't, the session comes up anyway, with no authentication at all. The only trace is in FRR's log: Unable to set TCP MD5 option… Protocol not available.

If you configure BGP passwords, check that they're in force rather than assuming it, especially on containers and unusual platforms: search the log for MD5 errors after every change. TCP-AO, the modern replacement, has the same dependency on kernel support.

r1: remote-as 64999 for a peer in AS 64501 (real output)
r1# show bgp summary

IPv4 Unicast Summary:
BGP router identifier 198.51.100.241, local AS number 64500 VRF default vrf-id 0
BGP table version 9
RIB entries 3, using 384 bytes of memory
Peers 1, using 17 KiB of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
192.0.2.1       4      64999         9        10        0    0    0 00:00:12         Idle        0 r2

Total number of neighbors 1

r1:~# grep -E "NOTIFICATION|bad OPEN|Bad Peer|ADJCHANGE" /var/log/frr/frr.log | tail -4
2026/10/18 03:34:06 BGP: [HZN6M-XRM1G] %NOTIFICATION: sent to neighbor 192.0.2.1 2/2 (OPEN Message Error/Bad Peer AS) 2 bytes f
2026/10/18 03:34:06 BGP: [HZN6M-XRM1G] %NOTIFICATION: sent to neighbor 192.0.2.1 2/2 (OPEN Message Error/Bad Peer AS) 2 bytes f
2026/10/18 03:34:10 BGP: [HZN6M-XRM1G] %NOTIFICATION: sent to neighbor 192.0.2.1 2/2 (OPEN Message Error/Bad Peer AS) 2 bytes f
2026/10/18 03:34:10 BGP: [HZN6M-XRM1G] %NOTIFICATION: sent to neighbor 192.0.2.1 2/2 (OPEN Message Error/Bad Peer AS) 2 bytes f

r1:~# vtysh -c "show bgp neighbors 192.0.2.1" | grep -E "BGP state|Last reset|Connections|Message"
  BGP state = Idle
  Message statistics:
  Connections established 1; dropped 1
  Last reset 00:00:12,  Notification sent (OPEN Message Error/Bad Peer AS)
  Message received that caused BGP to send a NOTIFICATION:

r2:~# grep -E "NOTIFICATION|ADJCHANGE" /var/log/frr/frr.log | tail -3
2026/10/18 03:34:10 BGP: [HZN6M-XRM1G] %NOTIFICATION: sent to neighbor 192.0.2.0 6/7 (Cease/Connection Collision Resolution) 0 bytes 
2026/10/18 03:34:10 BGP: [HZN6M-XRM1G] %NOTIFICATION: sent to neighbor 192.0.2.0 6/7 (Cease/Connection Collision Resolution) 0 bytes 
2026/10/18 03:34:10 BGP: [HZN6M-XRM1G] %NOTIFICATION: received from neighbor 192.0.2.0 2/2 (OPEN Message Error/Bad Peer AS) 2 bytes fb f5
  • Line 10: Back to Idle after each failed attempt; the AS column shows what r1 expects, 64999.
  • Lines 15–18: Every attempt ends the same way: r1 sends NOTIFICATION 2/2, Bad Peer AS.
  • Line 24: The last reset reason, in plain words.
  • Line 30: r2's side: it received the NOTIFICATION, so the configuration to fix is r1's.
r1 ↔ r3 between loopbacks, without ebgp-multihop (real output)
r1:~# ping -c 3 -I 198.51.100.241 198.51.100.243
PING 198.51.100.243 (198.51.100.243) from 198.51.100.241: 56 data bytes
64 bytes from 198.51.100.243: seq=0 ttl=63 time=0.073 ms
64 bytes from 198.51.100.243: seq=1 ttl=63 time=0.211 ms
64 bytes from 198.51.100.243: seq=2 ttl=63 time=0.322 ms

--- 198.51.100.243 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.073/0.202/0.322 ms

r1# show bgp summary

IPv4 Unicast Summary:
BGP router identifier 198.51.100.241, local AS number 64500 VRF default vrf-id 0
BGP table version 5
RIB entries 9, using 1152 bytes of memory
Peers 2, using 33 KiB of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
192.0.2.1       4      64501         7         5        5    0    0 00:00:30            4        1 r2
198.51.100.243  4      64501         0         0        0    0    0    never       Active        0 r3 loopback

Total number of neighbors 2

r1:~# vtysh -c "show bgp neighbors 198.51.100.243" | grep -E "BGP state|External BGP|Connections"
  BGP state = Active
  Connections established 0; dropped 0
  External BGP neighbor may be up to 1 hops away.
  • Lines 1–8: The loopbacks reach each other fine (a ping uses the normal TTL of 64).
  • Line 21: Yet the session never comes up.
  • Line 28: The clue: eBGP assumes the neighbour is one hop away.
r2: tcpdump of r1's SYN to r3 (real output)
r2:~# timeout 5 tcpdump -lnnvi eth0 "(tcp port 179 and host 198.51.100.243) or icmp"
tcpdump: listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
03:45:32.054239 IP (tos 0xc0, ttl 1, id 48392, offset 0, flags [DF], proto TCP (6), length 60)
    198.51.100.241.38579 > 198.51.100.243.179: Flags [S], cksum 0x567a (incorrect -> 0x57e6), seq 4047039902, win 64240, options [mss 1460,sackOK,TS val 1777974946 ecr 0,nop,wscale 10], length 0
03:45:32.054263 IP (tos 0xc0, ttl 64, id 3180, offset 0, flags [none], proto ICMP (1), length 88)
    192.0.2.1 > 198.51.100.241: ICMP time exceeded in-transit, length 68
	IP (tos 0xc0, ttl 1, id 48392, offset 0, flags [DF], proto TCP (6), length 60)
    198.51.100.241.38579 > 198.51.100.243.179: Flags [S], cksum 0x567a (incorrect -> 0x57e6), seq 4047039902, win 64240, options [mss 1460,sackOK,TS val 1777974946 ecr 0,nop,wscale 10], length 0
2 packets captured
2 packets received by filter
0 packets dropped by kernel
  • Lines 3–4: r1's SYN arrives at r2 with ttl 1: eBGP's default.
  • Lines 5–6: r2 can't forward a packet whose TTL would reach zero, so it sends back ICMP time exceeded. The SYN never reaches r3.
Both ends: ebgp-multihop 2, then clear (real output)
r1# clear bgp 198.51.100.243

r3# clear bgp 198.51.100.241

r1:~# vtysh -c "show bgp summary" | grep -E "^Neighbor|^198.51"
Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
198.51.100.243  4      64501         0         0        0    0    0    never       Active        0 r3 loopback

r1# clear bgp 198.51.100.243

r1# show bgp summary

IPv4 Unicast Summary:
BGP router identifier 198.51.100.241, local AS number 64500 VRF default vrf-id 0
BGP table version 5
RIB entries 9, using 1152 bytes of memory
Peers 2, using 33 KiB of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
192.0.2.1       4      64501         7         5        5    0    0 00:00:52            4        1 r2
198.51.100.243  4      64501         3         3        5    0    0 00:00:07     (Policy) (Policy) r3 loopback

Total number of neighbors 2

r1:~# vtysh -c "show bgp neighbors 198.51.100.243" | grep -E "BGP state|External BGP|Local host|Foreign host"
  BGP state = Established, up for 00:00:07
  External BGP neighbor may be up to 2 hops away.
Local host: 198.51.100.241, Local port: 36809
Foreign host: 198.51.100.243, Foreign port: 179
  • Lines 1–3: After neighbor … ebgp-multihop 2 on both ends, clear both sides.
  • Line 7: In this lab the first clear, straight after the change, wasn't enough: still Active 15 seconds later.
  • Line 9: A second clear a few seconds later brings it up. If a fix doesn't seem to work, clear again before deciding it's wrong.
  • Line 21: Established, but with (Policy): this new eBGP session has no route-maps yet (see the next section).
  • Line 27: Now allowed up to 2 hops.
r1: a password on r1 only, on Docker Desktop (real output)
r1# show bgp summary

IPv4 Unicast Summary:
BGP router identifier 198.51.100.241, local AS number 64500 VRF default vrf-id 0
BGP table version 13
RIB entries 9, using 1152 bytes of memory
Peers 1, using 17 KiB of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
192.0.2.1       4      64501        13        12       13    0    0 00:00:10            4        1 r2

Total number of neighbors 1

r1:~# grep -E "MD5|setsockopt" /var/log/frr/frr.log | tail -3
2026/10/18 03:42:55 BGP: [NWGVJ-FEW9F][EC 33554495] Unable to set TCP MD5 option on socket for peer 192.0.2.1 (sock=23): Protocol not available
2026/10/18 03:42:55 BGP: [KTTWK-0CPJ7][EC 100663303] sockopt_tcp_signature: setsockopt(26): Protocol not available
2026/10/18 03:42:55 BGP: [NWGVJ-FEW9F][EC 33554495] Unable to set TCP MD5 option on socket for peer 192.0.2.1 (sock=26): Protocol not available

r1:~# zcat /proc/config.gz | grep TCP_MD5SIG
# CONFIG_TCP_MD5SIG is not set
  • Line 10: Established, 4 prefixes, even though r2 has no password. On a real router this session couldn't form.
  • Lines 15–17: The only clue: FRR couldn't apply the password at all.
  • Lines 19–20: Why: this kernel was built without TCP-MD5.

Check your understanding

Router A logs NOTIFICATION: sent to neighbor B 2/2 (OPEN Message Error/Bad Peer AS). Router B logs that it received it. Whose configuration is most likely wrong?

  • A's: A expected a different AS from the one B announced in its OPEN.
  • B's: it sent the wrong AS.
  • Neither: 2/2 is a TCP error.
  • Both have mismatched passwords.