Troubleshooting

Strict reverse-path filtering

rp_filter makes the kernel check, for each incoming packet, that the route back to its source leaves through the interface it arrived on. In strict mode (1), a common hardening setting, that breaks a dual-ISP router: a packet from the internet arriving on ISP 2 fails the check, because the main table's route back to it goes via ISP 1.

The lab shows how subtle it is. With strict mode on, pings to the router's ISP 2 address still work (for a packet to the router itself, the reverse check uses the router's address as the source, and the source rule routes that via ISP 2), but the port forward through ISP 2 fails. Use loose mode (2), which only checks that the source is reachable through some interface, or set net.ipv4.conf.all.src_valid_mark = 1 so the check takes marks into account.

Common problems

Symptom	Likely cause	Check
Inbound to ISP 2's address doesn't work	Replies leave via ISP 1 and are dropped by its source filter	ip route get <client> from 203.0.113.2; the from rules
Router's ISP 2 address works, port forwards through ISP 2 don't	Missing connection marks, or rp_filter strict	conntrack -L (mark=2?); sysctl net.ipv4.conf.all.rp_filter
Office can't reach its own LAN or the ISP links after adding rules	A rule sends local traffic to an ISP table	The suppress_prefixlength 0 rule must come first
No failover although the internet is unreachable	The check only tests the link or the ISP's router	Probe targets beyond the ISP, from each ISP's address
Failover happens, but connections hang for minutes	Old conntrack entries still NATed to the dead ISP	conntrack -D --reply-dst <old address> on switch
Flapping between ISPs	Thresholds too low, or a single flaky target	More checks before switching back; several targets
Everything breaks after a reboot	Rules, tables or timer not persistent	systemctl is-enabled dualwan-routes dualwan-check.timer nftables

Using both at once

This lesson uses ISP 2 only as a backup. Linux can also spread new connections across both, with a multipath default route (ip route replace default nexthop via 198.51.100.1 dev eth0 nexthop via 203.0.113.1 dev eth1). By default the kernel picks the ISP per pair of source and destination addresses, so a given site always sees one address, but the health check has to remove a failed ISP's next hop, and some websites object to one user's connections arriving from two addresses. Failover alone is simpler, and what most offices need.

root@gw:~# sysctl -w net.ipv4.conf.all.rp_filter=1
net.ipv4.conf.all.rp_filter = 1

root@web:~# ping -c 2 -W 1 203.0.113.2 | tail -2; curl -s -o /dev/null -w "%{http_code}\n" -m 3 http://203.0.113.2:8080/
2 packets transmitted, 2 received, 0% packet loss, time 1028ms
rtt min/avg/max/mdev = 0.066/0.071/0.076/0.005 ms
000

root@gw:~# sysctl -w net.ipv4.conf.all.rp_filter=2
net.ipv4.conf.all.rp_filter = 2

root@web:~# ping -c 2 -W 1 203.0.113.2 | tail -2; curl -s -o /dev/null -w "%{http_code}\n" -m 3 http://203.0.113.2:8080/
2 packets transmitted, 2 received, 0% packet loss, time 1014ms
rtt min/avg/max/mdev = 0.077/0.078/0.080/0.001 ms
200