Management on every router
The VyOS with basic BGP lesson locked down a single router: SSH bound to a management address, and an input firewall that only accepts SSH from the management network and BGP from configured neighbors. Every router here needs the same, with SSH bound to its loopback. But an internal router also has to accept the protocols that hold the network together, and those are easy to forget.
What happens if you forget OSPF
Apply the single-router policy to core1 unchanged: SSH from the management LAN, BGP from the three loopbacks, and drop everything else. It commits without complaint. About 40 seconds later:
- Both OSPF adjacencies are gone. OSPF hellos are just packets addressed to the router (to the multicast address 224.0.0.5), and the default drop discards them. The
defaultcounter shows them piling up. Once no hello has arrived for the dead interval, each adjacency drops. - The loopbacks become unreachable. core1's route to edge1's loopback now falls through to its own blackhole for the /24.
- iBGP is next. The sessions still say Established for now, but only because their hold timers haven't expired, and every route that depends on a loopback next hop is already unusable.
- Traffic stops. The LAN host can no longer reach the internet.
One more rule fixes it: accept the OSPF protocol (IP protocol 89). Within seconds both adjacencies are back, and the new rule's counter shows the hellos arriving. This outage was shorter than the 180-second BGP hold time, so the iBGP sessions survived. A longer one would have taken them down too.
The policy for every router
| Rule | Why |
|---|---|
| Established and related; drop invalid | Replies to connections the router opened |
Loopback interface (lo) | The router talking to itself |
| ICMP | Ping and path MTU discovery |
| SSH from the management networks | Management, bound to the loopback |
| BGP from the other routers' loopbacks | iBGP is sourced from loopbacks, not link addresses |
| BGP from external neighbors (edges only) | Transit and IX sessions |
| OSPF | The IGP. Without it, everything else fails |
On the edges, add a second address group for the external BGP neighbors. With the full policy on all three routers, everything still works: OSPF and all three BGP sessions are up on edge1, the LAN host reaches the IX member, and SSH answers on the loopbacks from the management LAN but times out from the upstream. The upstream can still open TCP 179 to edge1, because it's a configured eBGP neighbor.
If you add IPv6 (see the OSPF on VyOS lesson for OSPFv3), the IPv6 input filter needs the same rules, including OSPF, which OSPFv3 also uses as its protocol number.
set service ssh listen-address '198.51.100.241'
set firewall group network-group MGMT-NETS network '198.51.100.0/25'
set firewall group address-group IBGP-PEERS address '198.51.100.241'
set firewall group address-group IBGP-PEERS address '198.51.100.242'
set firewall group address-group IBGP-PEERS address '198.51.100.243'
set firewall ipv4 input filter default-action 'drop'
set firewall ipv4 input filter rule 10 action 'accept'
set firewall ipv4 input filter rule 10 state 'established'
set firewall ipv4 input filter rule 10 state 'related'
set firewall ipv4 input filter rule 20 action 'drop'
set firewall ipv4 input filter rule 20 state 'invalid'
set firewall ipv4 input filter rule 30 action 'accept'
set firewall ipv4 input filter rule 30 inbound-interface name 'lo'
set firewall ipv4 input filter rule 40 action 'accept'
set firewall ipv4 input filter rule 40 protocol 'icmp'
set firewall ipv4 input filter rule 50 action 'accept'
set firewall ipv4 input filter rule 50 protocol 'tcp'
set firewall ipv4 input filter rule 50 destination port '22'
set firewall ipv4 input filter rule 50 source group network-group 'MGMT-NETS'
set firewall ipv4 input filter rule 60 action 'accept'
set firewall ipv4 input filter rule 60 protocol 'tcp'
set firewall ipv4 input filter rule 60 destination port '179'
set firewall ipv4 input filter rule 60 source group address-group 'IBGP-PEERS'- Line 1: SSH on core1's loopback.
- Lines 3–5: BGP will be accepted from all three loopbacks.
- Line 6: Drop anything not allowed below…
- Lines 7–23: …which, so far, doesn't include OSPF.
vyos@core1:~$ show ip ospf neighbor
Neighbor ID Pri State Up Time Dead Time Address Interface RXmtL RqstL DBsmL
vyos@core1:~$ show ip bgp summary
IPv4 Unicast Summary:
BGP router identifier 198.51.100.241, local AS number 64500 VRF default vrf-id 0
BGP table version 5
RIB entries 5, using 800 bytes of memory
Peers 2, using 60 KiB of memory
Peer groups 1, using 72 bytes of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
198.51.100.242 4 64500 7 6 5 0 0 00:02:41 2 2 edge1
198.51.100.243 4 64500 12 11 5 0 0 00:07:38 1 2 edge2
Total number of neighbors 2
vyos@core1:~$ show ip route 198.51.100.242
Routing entry for 198.51.100.0/24
Known via "static", distance 1, metric 0, best
Last update 00:07:41 ago
Flags: Recursion Selected RR Distance
Status: Installed
* unreachable (blackhole), weight 1
vyos@core1:~$ show firewall ipv4 input filter
Ruleset Information
---------------------------------
ipv4 Firewall "input filter"
Rule Action Protocol Packets Bytes Conditions
------- -------- ---------- --------- ------- --------------------------------------------
10 accept all 0 0 ct state { established, related } accept
20 drop all 0 0 ct state invalid
30 accept all 0 0 iifname "lo" accept
40 accept icmp 0 0 meta l4proto icmp accept
50 accept tcp 0 0 tcp dport 22 ip saddr @N_MGMT-NETS accept
60 accept tcp 0 0 tcp dport 179 ip saddr @A_IBGP-PEERS accept
default drop all 11 848 comment "INP-filter default-action "
root@host:~# ping -c 2 -W 2 203.0.113.130
PING 203.0.113.130 (203.0.113.130) 56(84) bytes of data.
--- 203.0.113.130 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1006ms- Line 3: No OSPF neighbors at all.
- Lines 15–16: The iBGP sessions only look healthy: their hold timers haven't expired yet.
- Lines 20–25: edge1's loopback now matches core1's own blackhole.
- Line 41: The dropped OSPF hellos.
- Lines 42–46: The LAN has lost the internet.
set firewall ipv4 input filter rule 70 action 'accept'
set firewall ipv4 input filter rule 70 protocol 'ospf'- Lines 1–2: Accept OSPF (IP protocol 89). Needed on every router running OSPF.
vyos@core1:~$ show ip ospf neighbor
Neighbor ID Pri State Up Time Dead Time Address Interface RXmtL RqstL DBsmL
198.51.100.242 1 Full/- 4.002s 35.972s 198.51.100.224 eth0:198.51.100.225 1 0 0
198.51.100.243 1 Full/- 4.000s 35.968s 198.51.100.226 eth1:198.51.100.227 1 0 0
vyos@core1:~$ show ip bgp summary
IPv4 Unicast Summary:
BGP router identifier 198.51.100.241, local AS number 64500 VRF default vrf-id 0
BGP table version 5
RIB entries 5, using 800 bytes of memory
Peers 2, using 60 KiB of memory
Peer groups 1, using 72 bytes of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
198.51.100.242 4 64500 7 7 5 0 0 00:03:21 2 2 edge1
198.51.100.243 4 64500 12 12 5 0 0 00:08:18 1 2 edge2
Total number of neighbors 2
vyos@core1:~$ show firewall ipv4 input filter
Ruleset Information
---------------------------------
ipv4 Firewall "input filter"
Rule Action Protocol Packets Bytes Conditions
------- -------- ---------- --------- ------- --------------------------------------------
10 accept all 0 0 ct state { established, related } accept
20 drop all 0 0 ct state invalid
30 accept all 0 0 iifname "lo" accept
40 accept icmp 0 0 meta l4proto icmp accept
50 accept tcp 0 0 tcp dport 22 ip saddr @N_MGMT-NETS accept
60 accept tcp 0 0 tcp dport 179 ip saddr @A_IBGP-PEERS accept
70 accept ospf 17 1960 meta l4proto ospf accept
default drop all 0 0 comment "INP-filter default-action "
root@host:~# ping -c 2 -W 2 203.0.113.130
PING 203.0.113.130 (203.0.113.130) 56(84) bytes of data.
64 bytes from 203.0.113.130: icmp_seq=1 ttl=62 time=2.20 ms
64 bytes from 203.0.113.130: icmp_seq=2 ttl=62 time=4.86 ms
--- 203.0.113.130 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 2.197/3.530/4.863/1.333 ms- Lines 4–5: Both adjacencies back.
- Lines 17–18: The iBGP sessions survived: same uptimes as before.
- Line 35: The OSPF rule doing its job.
- Lines 39–40: Traffic flows again.
set service ssh listen-address '198.51.100.242'
set firewall group network-group MGMT-NETS network '198.51.100.0/25'
set firewall group address-group IBGP-PEERS address '198.51.100.241'
set firewall group address-group IBGP-PEERS address '198.51.100.242'
set firewall group address-group IBGP-PEERS address '198.51.100.243'
set firewall group address-group EBGP-PEERS address '203.0.113.0'
set firewall ipv4 input filter default-action 'drop'
set firewall ipv4 input filter rule 10 action 'accept'
set firewall ipv4 input filter rule 10 state 'established'
set firewall ipv4 input filter rule 10 state 'related'
set firewall ipv4 input filter rule 20 action 'drop'
set firewall ipv4 input filter rule 20 state 'invalid'
set firewall ipv4 input filter rule 30 action 'accept'
set firewall ipv4 input filter rule 30 inbound-interface name 'lo'
set firewall ipv4 input filter rule 40 action 'accept'
set firewall ipv4 input filter rule 40 protocol 'icmp'
set firewall ipv4 input filter rule 50 action 'accept'
set firewall ipv4 input filter rule 50 protocol 'tcp'
set firewall ipv4 input filter rule 50 destination port '22'
set firewall ipv4 input filter rule 50 source group network-group 'MGMT-NETS'
set firewall ipv4 input filter rule 60 action 'accept'
set firewall ipv4 input filter rule 60 protocol 'tcp'
set firewall ipv4 input filter rule 60 destination port '179'
set firewall ipv4 input filter rule 60 source group address-group 'IBGP-PEERS'
set firewall ipv4 input filter rule 61 action 'accept'
set firewall ipv4 input filter rule 61 protocol 'tcp'
set firewall ipv4 input filter rule 61 destination port '179'
set firewall ipv4 input filter rule 61 source group address-group 'EBGP-PEERS'
set firewall ipv4 input filter rule 70 action 'accept'
set firewall ipv4 input filter rule 70 protocol 'ospf'- Line 6: Its external BGP neighbor (edge2 has the IX route server).
- Lines 25–28: eBGP from that neighbor.
- Lines 29–30: And OSPF.
vyos@edge1:~$ show ip ospf neighbor
Neighbor ID Pri State Up Time Dead Time Address Interface RXmtL RqstL DBsmL
198.51.100.241 1 Full/- 2m21s 34.316s 198.51.100.225 eth1:198.51.100.224 0 0 0
198.51.100.243 1 Full/- 5m51s 38.108s 198.51.100.229 eth2:198.51.100.228 0 0 0
vyos@edge1:~$ show ip bgp summary
IPv4 Unicast Summary:
BGP router identifier 198.51.100.242, local AS number 64500 VRF default vrf-id 0
BGP table version 5
RIB entries 5, using 800 bytes of memory
Peers 3, using 90 KiB of memory
Peer groups 1, using 72 bytes of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
198.51.100.241 4 64500 9 10 5 0 0 00:05:40 2 2 core1
198.51.100.243 4 64500 9 10 5 0 0 00:05:40 1 2 edge2
203.0.113.0 4 64496 10 8 5 0 0 00:04:37 2 1 UPSTREAM transit
Total number of neighbors 3
vyos@edge1:~$ show firewall ipv4 input filter
Ruleset Information
---------------------------------
ipv4 Firewall "input filter"
Rule Action Protocol Packets Bytes Conditions
------- -------- ---------- --------- ------- --------------------------------------------
10 accept all 17 1373 ct state { established, related } accept
20 drop all 0 0 ct state invalid
30 accept all 12 720 iifname "lo" accept
40 accept icmp 0 0 meta l4proto icmp accept
50 accept tcp 0 0 tcp dport 22 ip saddr @N_MGMT-NETS accept
60 accept tcp 0 0 tcp dport 179 ip saddr @A_IBGP-PEERS accept
61 accept tcp 0 0 tcp dport 179 ip saddr @A_EBGP-PEERS accept
70 accept ospf 17 1156 meta l4proto ospf accept
default drop all 1 71 comment "INP-filter default-action "
root@host:~# nc -zv -w 3 198.51.100.241 22
Connection to 198.51.100.241 22 port [tcp/ssh] succeeded!
root@host:~# nc -zv -w 3 198.51.100.242 22
Connection to 198.51.100.242 22 port [tcp/ssh] succeeded!
root@host:~# ping -c 1 -W 2 203.0.113.65
PING 203.0.113.65 (203.0.113.65) 56(84) bytes of data.
64 bytes from 203.0.113.65: icmp_seq=1 ttl=62 time=2.14 ms
--- 203.0.113.65 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 2.138/2.138/2.138/0.000 ms
upstream:/# nc -zv -w 3 198.51.100.242 22
nc: 198.51.100.242 (198.51.100.242:22): Operation timed out
upstream:/# nc -zv -w 3 198.51.100.242 179
198.51.100.242 (198.51.100.242:179) open- Lines 4–5: OSPF up.
- Lines 17–19: iBGP to both loopbacks, and eBGP to the upstream.
- Lines 30–38: Each rule's counter shows what it's matching.
- Lines 39–42: SSH on the loopbacks from the management LAN.
- Lines 43–45: The LAN still reaches the IX member.
- Lines 50–51: SSH from the internet: dropped.
- Lines 52–53: BGP from the upstream: allowed, because it's a configured neighbor.
Check your understanding
After applying an input firewall to a core router, OSPF neighbors disappear about 40 seconds later. Why?
- OSPF hellos are packets addressed to the router, and the input filter's default drop discards them
- The firewall blocks multicast on the links
- BGP took the routes away
- SSH was bound to the loopback