TTR Networks

Intermediate · Two internet connections on VyOS · section 5 of 7

Port forwards on both ISPs

✓ Lab-tested 21 October 2026 on VyOS 2026.10.16-0024-rolling, Debian 12 hosts

The office's web server is published through both ISPs, with a port forward (destination NAT) on each WAN address. Through ISP1, it worked. Through ISP2, the server on the internet got no answer at all, and the web server never logged the request.

A capture on ISP1 shows why: the web server's replies, correctly translated back to ISP2's address 198.51.100.2, were leaving through ISP1, because R1's default route points there. ISP1 drops packets with a source address that isn't its customer's, so the replies never arrived. (Even an ISP that didn't filter would deliver replies from an address the client never contacted.)

The rule is: reply through the interface the connection came in on. sticky-connections inbound does exactly that. Connections arriving on eth0 or eth1 get that ISP's connection mark, and their replies are routed through the same ISP. With it, the web server answered through both.

You can see the whole mechanism in nftables:

  • Connections arriving on eth0 or eth1 are marked for that ISP.
  • New connections from the LAN (eth2) jump to the chain for the current ISP.
  • Every packet from the LAN copies its connection's mark into the packet mark.
  • Two policy rules send marked packets to a table with that ISP's default route.

It's the same technique as the full tunnel in WireGuard on Debian: marks and policy routing.

A port forward on each ISP's address
set nat destination rule 10 description 'Web server, via ISP1'
set nat destination rule 10 inbound-interface name 'eth0'
set nat destination rule 10 protocol 'tcp'
set nat destination rule 10 destination port '80'
set nat destination rule 10 translation address '10.0.0.10'
set nat destination rule 20 description 'Web server, via ISP2'
set nat destination rule 20 inbound-interface name 'eth1'
set nat destination rule 20 protocol 'tcp'
set nat destination rule 20 destination port '80'
set nat destination rule 20 translation address '10.0.0.10'
Through ISP1 it works; through ISP2 it doesn't (real output)
root@server:~# curl -s -m 5 http://203.0.113.2/ || echo "no answer"
hello from lanhost

root@server:~# curl -s -m 5 http://198.51.100.2/ || echo "no answer"
no answer

root@lanhost:~# tail -2 /tmp/http.log
192.0.2.10 - - [21/Oct/2026 09:37:05] "GET / HTTP/1.1" 200 -
  • Lines 4–5: No answer through ISP2.
  • Lines 7–8: Only the ISP1 request reached the web server.
Where the replies went (real output)
root@isp1:~# timeout 8 tcpdump -lnni eth0 tcp port 80   # while server tries http://198.51.100.2/
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
09:37:20.179485 IP 198.51.100.2.80 > 192.0.2.10.50634: Flags [S.], seq 1923695506, ack 1273436634, win 65160, options [mss 1460,sackOK,TS val 3568155222 ecr 4173658836,nop,wscale 10], length 0
09:37:21.196522 IP 198.51.100.2.80 > 192.0.2.10.50634: Flags [S.], seq 1923695506, ack 1273436634, win 65160, options [mss 1460,sackOK,TS val 3568156237 ecr 4173658836,nop,wscale 10], length 0
09:37:21.197866 IP 198.51.100.2.80 > 192.0.2.10.50634: Flags [S.], seq 1923695506, ack 1273436634, win 65160, options [mss 1460,sackOK,TS val 3568156240 ecr 4173658836,nop,wscale 10], length 0
09:37:22.227128 IP 198.51.100.2.80 > 192.0.2.10.50634: Flags [S.], seq 1923695506, ack 1273436634, win 65160, options [mss 1460,sackOK,TS val 3568157269 ecr 4173658836,nop,wscale 10], length 0
09:37:23.251556 IP 198.51.100.2.80 > 192.0.2.10.50634: Flags [S.], seq 1923695506, ack 1273436634, win 65160, options [mss 1460,sackOK,TS val 3568158293 ecr 4173658836,nop,wscale 10], length 0
09:37:24.017714 IP 198.51.100.2.80 > 192.0.2.10.38984: Flags [S.], seq 1582450202, ack 2700219387, win 65160, options [mss 1460,sackOK,TS val 354923371 ecr 4055941856,nop,wscale 10], length 0
09:37:24.269062 IP 198.51.100.2.80 > 192.0.2.10.50634: Flags [S.], seq 1923695506, ack 1273436634, win 65160, options [mss 1460,sackOK,TS val 3568159311 ecr 4173658836,nop,wscale 10], length 0
09:37:26.325138 IP 198.51.100.2.80 > 192.0.2.10.50634: Flags [S.], seq 1923695506, ack 1273436634, win 65160, options [mss 1460,sackOK,TS val 3568161367 ecr 4173658836,nop,wscale 10], length 0

8 packets captured
8 packets received by filter
0 packets dropped by kernel
  • Lines 4–11: The web server's replies, with ISP2's address, leaving through ISP1, which drops them. The server keeps retrying.
Reply the way the connection came in
set load-balancing wan sticky-connections inbound
Both work (real output)
root@server:~# curl -s -m 5 http://203.0.113.2/ || echo "no answer"
hello from lanhost

root@server:~# curl -s -m 5 http://198.51.100.2/ || echo "no answer"
hello from lanhost

root@lanhost:~# tail -2 /tmp/http.log
192.0.2.10 - - [21/Oct/2026 09:37:53] "GET / HTTP/1.1" 200 -
192.0.2.10 - - [21/Oct/2026 09:37:53] "GET / HTTP/1.1" 200 -
  • Lines 1–5: Answered through either ISP.
  • Lines 7–9: Both requests reached the web server.

Check your understanding

A connection arrives through ISP2. Without sticky connections, why does it fail even though ISP2 is healthy?

  • The replies follow R1's default route out through ISP1, carrying ISP2's address, and ISP1 drops them.
  • Destination NAT only works on the first interface.
  • ISP2 blocks inbound connections.
  • The web server only listens on ISP1's address.