A routing table per ISP
Linux has many routing tables, not one. ip route shows the main table; others are numbered, and a name can be given to each in /etc/iproute2/rt_tables.d/. Rules (ip rule) decide which table a packet uses, in order of preference, and can match on the source address, a firewall mark, the incoming interface and more.
The setup script gives each ISP a table containing just its default route, and adds four kinds of rule:
- Preference 900: local networks first.
lookup main suppress_prefixlength 0uses the main table, but ignores any route with a prefix length of 0, the default route. So traffic for the office LAN and the ISP links finds its connected route, whichever rule would otherwise match. It's the trickwg-quickuses for full tunnels. - 1001 and 1002: by source address. Anything leaving with ISP 1's address uses ISP 1's table, and the same for ISP 2. This covers the router's own traffic: replies to pings, the health check's probes, a VPN on the router.
- 1101 and 1102: by firewall mark. Forwarded traffic, such as the port forward's replies from the office server, has a LAN source address, so the next section marks it.
- Everything else falls through to the main table, whose default route says which ISP is currently in use. The health check will manage that route.
A small script plus a systemd oneshot service applies it all at boot, and running it again is safe.
Now the router answers on both addresses, and ip route get shows traffic from 203.0.113.2 leaving via ISP 2, while the LAN is still reached directly. But the port forward through ISP 2 still fails: the office server's replies come from 10.20.0.10, which no source rule matches.
# /etc/iproute2/rt_tables.d/dualwan.conf: names for the two extra routing tables.
101 wan1
102 wan2- Lines 2–3: Names for table numbers, so ip rule and ip route can say wan1 and wan2.
#!/bin/sh
# /usr/local/sbin/dualwan-routes: a routing table per ISP, and the rules that choose them.
# Run at boot by dualwan-routes.service. Safe to run again.
set -e
ip route replace default via 198.51.100.1 dev eth0 table wan1
ip route replace default via 203.0.113.1 dev eth1 table wan2
# Remove our rules first, so running it again doesn't add duplicates.
for p in 900 1001 1002 1101 1102; do ip rule del pref $p 2>/dev/null || true; done
# 1. Local networks first: use the main table, but ignore its default route.
ip rule add pref 900 lookup main suppress_prefixlength 0
# 2. Traffic from an ISP's address goes back out through that ISP.
ip rule add pref 1001 from 198.51.100.2 lookup wan1
ip rule add pref 1002 from 203.0.113.2 lookup wan2
# 3. Replies to connections that came in on an ISP (marked by nftables) go back the same way.
ip rule add pref 1101 fwmark 1 lookup wan1
ip rule add pref 1102 fwmark 2 lookup wan2
# Everything else uses the main table, whose default route the health check sets.- Lines 5–6: Each ISP's table holds just its default route.
- Line 9: Delete our rules first, so running the script again doesn't duplicate them.
- Line 11: Connected networks from the main table, ignoring its default route.
- Lines 13–14: Traffic from an ISP's address goes out through that ISP.
- Lines 16–17: Packets marked by nftables (next section) too.
# /etc/systemd/system/dualwan-routes.service
[Unit]
Description=Routing tables and rules for two ISPs
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/local/sbin/dualwan-routes
[Install]
WantedBy=multi-user.target- Lines 8–9: Run once at boot, and count as active afterwards.
root@gw:~# systemctl enable --now dualwan-routes && ip rule && ip route show table wan1 && ip route show table wan2
Created symlink '/etc/systemd/system/multi-user.target.wants/dualwan-routes.service' → '/etc/systemd/system/dualwan-routes.service'.
0: from all lookup local
900: from all lookup main suppress_prefixlength 0
1001: from 198.51.100.2 lookup wan1
1002: from 203.0.113.2 lookup wan2
1101: from all fwmark 0x1 lookup wan1
1102: from all fwmark 0x2 lookup wan2
32766: from all lookup main
32767: from all lookup default
default via 198.51.100.1 dev eth0
default via 203.0.113.1 dev eth1
root@gw:~# ip route get 192.0.2.80 from 203.0.113.2; ip route get 10.20.0.10 from 203.0.113.2
192.0.2.80 from 203.0.113.2 via 203.0.113.1 dev eth1 table wan2 uid 0
cache
10.20.0.10 from 203.0.113.2 dev eth2 uid 0
cache
root@web:~# ping -c 2 -W 1 203.0.113.2 | tail -2; curl -s -o /dev/null -w "%{http_code}\n" -m 3 http://203.0.113.2:8080/
2 packets transmitted, 2 received, 0% packet loss, time 1009ms
rtt min/avg/max/mdev = 0.070/0.074/0.079/0.004 ms
000- Lines 3–10: Rules are tried by preference, lowest first. 0, 32766 and 32767 are always there.
- Lines 11–12: ISP 1's table, then ISP 2's.
- Line 15: From ISP 2's address: via ISP 2, table wan2.
- Line 17: But the office LAN is still reached directly, thanks to rule 900.
- Lines 21–22: The router's ISP 2 address answers now…
- Line 23: …but the port forward through ISP 2 still fails.
Check your understanding
Why does rule 900 (lookup main suppress_prefixlength 0) come before the ISP rules?
- So traffic for connected networks, like the office LAN, uses its direct route instead of an ISP's default route.
- To make the main table's default route win.
- Because the kernel requires the main table first.
- To speed up lookups.