Two ISPs, four problems
Add a second ISP to a VyOS office router and make it actually work: why a backup default route misses most outages, health checks that test beyond the ISP, per-ISP NAT, port forwards through both ISPs, and what happens to open connections, all measured on a real lab.
By the end you will be able to:
- Explain why a floating default route doesn't fail over when the ISP's upstream breaks
- Configure VyOS WAN load balancing in failover mode, with health checks that test beyond each ISP
- Publish a service through both ISPs, with replies leaving the way the connection came in
- Predict what happens to existing connections on failover, and choose whether to flush them
- Read the connection marks and policy routing that VyOS generates
Prerequisites: Setting up VyOS with NAT
After a VPN, the most common request in a small office is a second internet connection, "so we're not offline when the line goes down". Doing it properly takes more than a second default route, and this lesson shows why with real failures on a VyOS router.
The lab
R1 has two ISPs. ISP1 (203.0.113.0/30) is the main line, and ISP2 (198.51.100.0/30) the backup. Each ISP is a separate router with its own uplink to the internet, where a server logs the address of everyone who connects. The office LAN (10.0.0.0/24) has one host, which also runs a web server that should be reachable from the internet through both ISPs.
Like real ISPs, both check the source address of everything their customer sends ("ingress filtering", BCP 38): a packet that leaves through ISP1 carrying ISP2's address is dropped. That one realistic detail is behind most of the surprises in this lesson.
Four problems to solve
- Detect a failure, including one where the link to the ISP stays up.
- Fail over: send new traffic through the other ISP, with source NAT to that ISP's address.
- Inbound: a connection that arrives through an ISP must be answered through the same ISP.
- Existing connections: decide what happens to them, because they can't simply move.
The same problems, solved piece by piece on a plain Linux router, are in Two internet connections on Debian.
- Normal: everything via ISP1. The LAN's traffic leaves through ISP1, translated to 203.0.113.2.
- ISP1 loses its upstream. R1's link to ISP1 stays up and ISP1's router still answers pings, so a backup default route never takes over.
- Health check beyond the ISP. R1 pings a target on the internet through each ISP. ISP1's fails, so new connections leave through ISP2, translated to 198.51.100.2.
- Inbound through ISP2. A connection that arrives through ISP2 must be answered through ISP2. sticky-connections inbound marks it, and its replies follow the mark.
#!/bin/sh
# Address the hosts. Each ISP is a router with its own uplink to the "internet" network, and
# checks the source address of everything its customer sends (strict reverse-path filtering, as BCP 38 asks).
cd "$(dirname "$0")"
x() { docker compose exec -T "$@"; }
for c in isp1 isp2 server lanhost; do x $c sh -c 'sysctl -qw net.ipv6.conf.all.disable_ipv6=1'; done
x isp1 sh -c 'sysctl -qw net.ipv4.ip_forward=1 net.ipv4.conf.all.rp_filter=1 net.ipv4.conf.eth0.rp_filter=1
ip addr add 203.0.113.1/30 dev eth0; ip addr add 192.0.2.1/24 dev eth1; ip route add 198.51.100.0/30 via 192.0.2.2'
x isp2 sh -c 'sysctl -qw net.ipv4.ip_forward=1 net.ipv4.conf.all.rp_filter=1 net.ipv4.conf.eth0.rp_filter=1
ip addr add 198.51.100.1/30 dev eth0; ip addr add 192.0.2.2/24 dev eth1; ip route add 203.0.113.0/30 via 192.0.2.1'
# The server: a web server, plus two addresses that R1 uses as health-check targets, one per ISP.
# The internet reaches each ISP's customers through that ISP.
x server sh -c 'ip addr add 192.0.2.10/24 dev eth0; ip addr add 192.0.2.101/24 dev eth0; ip addr add 192.0.2.102/24 dev eth0; ip route add 203.0.113.0/30 via 192.0.2.1; ip route add 198.51.100.0/30 via 192.0.2.2'
x lanhost sh -c 'ip addr add 10.0.0.10/24 dev eth0; ip route add default via 10.0.0.1'
# Web servers that log who connected: on the internet, and on the LAN (for the port forward).
for c in server lanhost; do x -d $c sh -c "cd /tmp && echo 'hello from $c' > index.html && exec python3 -m http.server 80 > /tmp/http.log 2>&1"; done
echo hosts ready- Lines 7–10: Each ISP: routing on, and strict reverse-path filtering on the customer interface, so packets with someone else's source address are dropped.
- Lines 11–13: The internet server, with two extra addresses used as health-check targets.
- Lines 15–16: Web servers that log every visitor.