Reboots, maintenance and troubleshooting

After a reboot

Everything is a systemd service, so a reboot of either router should bring it back into the pair without help. r2, restarted, was up again within 25 seconds: networkd, nftables, keepalived and conntrackd all running, and both VRRP instances in BACKUP behind r1. One thing a reboot loses: a restarted conntrackd only learns about connections as they change, so long-lived connections that were already open stay unknown to it. FTFW's StartupResync on option (in the Mode FTFW block) asks the other router for a complete copy at startup; this lab's configuration doesn't set it, so it wasn't tested here.

Planned maintenance

To update and reboot the master without disruption, make it give up the addresses first: systemctl stop keepalived on it. keepalived sends a final advertisement with priority 0, so the backup takes over at once instead of after 3.6 seconds, and conntrackd has already copied the connections. Update and reboot it, and let preemption (or you) move the addresses back.

Troubleshooting

Symptom	Check	Usual cause
Both routers hold the shared address	ip -br addr on both; tcpdump -ni staff vrrp	VRRP (protocol 112) blocked, VLAN missing on the switch, different virtual router IDs
Neither router holds it	journalctl -u keepalived	Both in FAULT: the health check fails on both, or the interface named in the config doesn't exist
LAN address on one router, WAN address on the other	ip -br addr on both	No vrrp_sync_group
Failover works, but open connections break	conntrack -L on the backup	conntrackd not syncing: not configured (Debian's default only counts), sync port blocked, or wrong addresses
conntrackd warns getprotobyname() cannot find protocol	ls /etc/protocols	The netbase package is missing, so the filter matches nothing and nothing is synced
Connections still break, though they're synced	conntrack -L on the backup: the reply address	NAT with masquerade (the old router's own address) instead of snat to the shared address
A second outage when the master returns	journalctl -u keepalived	Preemption: use nopreempt or preempt_delay if you'd rather choose when
The routers keep swapping	The health check's log lines	A check that fails on its own: test something reliable, and raise fall and rise

Cheat sheets: nftables, Network error messages explained.

root@r2:~# ps -o etime=,comm= -p 1; systemctl is-active systemd-networkd nftables keepalived conntrackd; journalctl -u keepalived --no-pager -o cat | grep -E "STATE" | tail -2
      00:25 systemd
active
active
active
active
(STAFF) Entering BACKUP STATE
(WAN) Entering BACKUP STATE