Failover, measured
Time to break something. While the office PC pings the internet every half second, ISP 1 fails beyond its router: in the lab, it silently stops forwarding anything.
From the router, at first, everything about ISP 1 looks fine: the link is up, and ISP 1's router answers pings. Only a ping through ISP 1 to the internet shows the problem. A check of the link or the next hop would never have failed over.
The health check saw three failures in a row and switched the default route to ISP 2 18 seconds after the outage began, clearing the 11 tracked flows NATed to ISP 1's address (the PC's ping among them). The office PC lost 36 pings, exactly those 18 seconds. Afterwards, the web server sees the office arriving from ISP 2's address.
When ISP 1 was repaired, the router waited for 6 good checks and moved back about 30 seconds later. This time the PC lost nothing: ISP 2 was still working while the switch happened.
Where the 18 seconds went
Detection is roughly checks needed × interval, plus the time a failing check takes (each probe waits a second for its answer): 3 × 5 s, plus a few seconds. To fail over faster, check more often or need fewer failures, at the cost of more false alarms. A few probes per second is fine for one router. Sub-second failover needs a different tool, such as BFD between routers.
What a failover can't fix
- Existing connections end: downloads, SSH sessions and calls restart, because the office now has a different public address.
- Inbound services on the failed ISP's address are unreachable until it's back. To keep them reachable, publish them through both ISPs (as the port forward here is) and have DNS give out both addresses, or a short TTL you can change.
- Anything that allow-lists the office's address (a supplier's firewall, a cloud admin portal) needs both addresses on its list.
root@isp1:~# date +%T; nft add table ip outage; nft add chain ip outage upstream "{ type filter hook forward priority 0; policy drop; }"
08:09:52
root@gw:~# ip -br link show eth0; ping -c 1 -W 1 198.51.100.1 | tail -2; ping -c 1 -W 1 -I 198.51.100.2 192.0.2.80 | tail -2
eth0@if7959 UP 32:6c:33:a7:7f:b0 <BROADCAST,MULTICAST,UP,LOWER_UP>
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.039/0.039/0.039/0.000 ms
1 packets transmitted, 0 received, 100% packet loss, time 0ms
root@gw:~# journalctl -t dualwan --no-pager -o short-iso | tail -2; ip route show default
2026-10-21T08:10:10+00:00 gw dualwan[694]: isp1 failed 3 checks in a row
2026-10-21T08:10:10+00:00 gw dualwan[699]: now using isp2 (cleared 11 flow entries)
default via 203.0.113.1 dev eth1
root@pc:~# curl -s -o /dev/null -w "%{http_code}\n" -m 3 http://192.0.2.80/
200
root@web:~# tail -1 /tmp/web.log
203.0.113.2 - - [21/Oct/2026 08:10:25] "GET / HTTP/1.1" 200 -- Lines 1–2: 08:09:52: ISP 1 silently stops forwarding.
- Lines 5–7: The link is up, and ISP 1's router answers…
- Line 8: …but nothing gets through it.
- Lines 12–14: 18 seconds later: three failed checks, a switch to ISP 2, and 11 stale flows cleared.
- Lines 16–20: The office is back, now as ISP 2's address.
root@pc:~# timeout 100 ping -O -i 0.5 192.0.2.80
PING 192.0.2.80 (192.0.2.80) 56(84) bytes of data.
64 bytes from 192.0.2.80: icmp_seq=1 ttl=61 time=0.105 ms
64 bytes from 192.0.2.80: icmp_seq=2 ttl=61 time=0.387 ms
64 bytes from 192.0.2.80: icmp_seq=3 ttl=61 time=0.091 ms
64 bytes from 192.0.2.80: icmp_seq=4 ttl=61 time=0.088 ms
64 bytes from 192.0.2.80: icmp_seq=5 ttl=61 time=0.118 ms
64 bytes from 192.0.2.80: icmp_seq=6 ttl=61 time=0.373 ms
64 bytes from 192.0.2.80: icmp_seq=7 ttl=61 time=0.297 ms
64 bytes from 192.0.2.80: icmp_seq=8 ttl=61 time=0.345 ms
64 bytes from 192.0.2.80: icmp_seq=9 ttl=61 time=0.317 ms
64 bytes from 192.0.2.80: icmp_seq=10 ttl=61 time=0.153 ms
no answer yet for icmp_seq=11
no answer yet for icmp_seq=12
[... 32 lines trimmed: no answer for icmp_seq 13 to 44 ...]
no answer yet for icmp_seq=45
no answer yet for icmp_seq=46
64 bytes from 192.0.2.80: icmp_seq=47 ttl=61 time=0.116 ms
64 bytes from 192.0.2.80: icmp_seq=48 ttl=61 time=0.089 ms
64 bytes from 192.0.2.80: icmp_seq=49 ttl=61 time=0.075 ms
64 bytes from 192.0.2.80: icmp_seq=50 ttl=61 time=0.082 ms
[... 146 lines trimmed: icmp_seq 51 to 196 all answered, through ISP 2 and then through ISP 1 again after the switch back at 08:11:01 ...]
64 bytes from 192.0.2.80: icmp_seq=197 ttl=61 time=0.367 ms
64 bytes from 192.0.2.80: icmp_seq=198 ttl=61 time=0.440 ms- Lines 3–12: Every half second, through ISP 1.
- Lines 13–17: The outage: 36 unanswered pings, 18 seconds.
- Lines 18–21: Through ISP 2.
- Lines 22–23: Nothing lost when the router switched back.
root@isp1:~# date +%T; nft delete table ip outage
08:10:30
root@gw:~# journalctl -t dualwan --no-pager -o short-iso | tail -1; ip route show default
2026-10-21T08:11:01+00:00 gw dualwan[815]: now using isp1 (cleared 2 flow entries)
default via 198.51.100.1 dev eth0 - Lines 1–2: 08:10:30: ISP 1 works again.
- Lines 4–6: Six good checks later (31 s), back on ISP 1.
Check your understanding
Which change would make this router fail over faster?
- Run the check more often, or switch after fewer failures.
- Raise RECOVER_AFTER.
- Add a third probe target.
- Lower the ping timeout to 0.1 s.
Check your understanding
After a failover, an office user's long download stops and has to be restarted. Why?
- The connection was NATed to ISP 1's address; through ISP 2 the office has a different public address, so the server can't continue it.
- The health check blocks downloads during a failover.
- ISP 2 is slower.
- The connection mark was lost.