Failover, and the DNS gap
Now the test that matters: R1 loses power. In the lab, its virtual machine is killed outright while pc1 pings the web server five times a second, and pc2 captures ARP.
- 3.5 seconds later, R2 announces itself: gratuitous ARP for 10.0.0.1 from R2's own MAC address, repeated so that every host and switch updates its tables. (VyOS uses the router's real MAC with gratuitous ARP. VRRP can also use a shared virtual MAC, but this is VyOS's default.)
- pc1 lost 16 pings, about 3.2 seconds: VRRP's default of three missed one-second advertisements, plus a little.
- R2 is now master of both groups, thanks to the sync group.
…and the DNS problem
Routing recovered in seconds. Name lookups didn't. Immediately after the failover, curl http://server.example.net/ failed: Resolving timed out after 5000 milliseconds. pc1's first DNS server is 10.0.0.2, which is dead. Its resolver waits five seconds, retries, and only then tries 10.0.0.3. A single dig took 15 seconds. Every application that looks up a name hangs, so for users the network is still down.
This is the classic gap in "redundant" office networks: the gateway fails over and DNS doesn't.
The fix: DNS on the virtual address too
Make each router's DNS forwarder also listen on 10.0.0.1, and hand out only 10.0.0.1 as the name server. Then DNS fails over exactly when the gateway does.
Can a backup router listen on an address it doesn't hold? On VyOS, yes. R1 was rebuilt with the DNS change before its VRRP configuration, while 10.0.0.1 belonged to R2, and its forwarder was already listening on 10.0.0.1:53 (VyOS lets it bind to a non-local address). So it's ready the moment the address arrives.
With VRRP back on R1, it preempted: priority 200 beats R2's 100, so R1 took both groups back. A planned handover like that lost no pings at all, unlike a crash. (If you'd rather avoid even a planned move when the primary returns, VRRP preemption can be turned off or delayed.)
Then the proof: pc1 renewed its lease (and now uses 10.0.0.1 for DNS), and R1 was killed again. Straight after the failover, a lookup took 29 milliseconds, answered by R2 at 10.0.0.1, and the web request worked first time.
root@lab:~# date +"killing r1 at %T.%N"; pkill -f console-r1.sock
killing r1 at 14:01:02.894266507
root@pc2:~# timeout 25 tcpdump -lnei eth0 arp
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
14:01:06.358112 52:54:00:02:00:01 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 10.0.0.1 (ff:ff:ff:ff:ff:ff) tell 10.0.0.1, length 28
14:01:06.358137 52:54:00:02:00:01 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 10.0.0.1 (ff:ff:ff:ff:ff:ff) tell 10.0.0.1, length 28
14:01:06.358139 52:54:00:02:00:01 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 10.0.0.1 (ff:ff:ff:ff:ff:ff) tell 10.0.0.1, length 28
14:01:06.358142 52:54:00:02:00:01 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 10.0.0.1 (ff:ff:ff:ff:ff:ff) tell 10.0.0.1, length 28
14:01:06.358146 52:54:00:02:00:01 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 10.0.0.1 (ff:ff:ff:ff:ff:ff) tell 10.0.0.1, length 28
14:01:06.365161 52:54:00:02:00:01 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 10.0.0.150 tell 10.0.0.3, length 28
14:01:11.358772 52:54:00:02:00:01 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 10.0.0.1 (ff:ff:ff:ff:ff:ff) tell 10.0.0.1, length 28
14:01:11.358863 52:54:00:02:00:01 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 10.0.0.1 (ff:ff:ff:ff:ff:ff) tell 10.0.0.1, length 28
14:01:11.358874 52:54:00:02:00:01 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 10.0.0.1 (ff:ff:ff:ff:ff:ff) tell 10.0.0.1, length 28
14:01:11.358879 52:54:00:02:00:01 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 10.0.0.1 (ff:ff:ff:ff:ff:ff) tell 10.0.0.1, length 28
14:01:11.358891 52:54:00:02:00:01 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 10.0.0.1 (ff:ff:ff:ff:ff:ff) tell 10.0.0.1, length 28
11 packets captured
11 packets received by filter
0 packets dropped by kernel
root@pc1:~# ping -D -i 0.2 -c 100 -W 1 192.0.2.80
[... 85 lines trimmed: the header and 84 replies; 16 replies are missing around the failover ...]
--- 192.0.2.80 ping statistics ---
100 packets transmitted, 84 received, 16% packet loss, time 20306ms
rtt min/avg/max/mdev = 0.579/1.959/4.329/0.750 ms
vyos@R2:~$ show vrrp
Name Interface VRID State Priority Last Transition
------ ----------- ------ ------- ---------- -----------------
LAN eth0 10 MASTER 100 20s
WAN eth1 20 MASTER 100 20s- Line 2: The moment R1 dies.
- Lines 6–10: 3.5 s later: R2 claims 10.0.0.1 with gratuitous ARP from its own MAC.
- Lines 12–16: Repeated five seconds later, to be sure.
- Line 25: 16 lost pings at 200 ms: about 3.2 s of outage.
- Lines 30–31: R2 is master of both groups.
root@pc1:~# curl -sS -m 5 http://server.example.net/
curl: (28) Resolving timed out after 5000 milliseconds
root@pc1:~# date +%T.%N; dig server.example.net +noall +answer +stats | grep -E "IN|SERVER"; date +%T.%N
14:01:56.278126962
server.example.net. 1 IN A 192.0.2.80
;; SERVER: 10.0.0.3#53(10.0.0.3) (UDP)
14:02:11.348424719- Line 2: Applications time out looking up names…
- Lines 4–7: …and a lookup takes 15 seconds, answered only by R2 after the dead R1 times out.
set service dns forwarding listen-address '10.0.0.1'
delete service dhcp-server shared-network-name LAN subnet 10.0.0.0/24 option name-server
set service dhcp-server shared-network-name LAN subnet 10.0.0.0/24 option name-server '10.0.0.1'- Line 1: Listen on 10.0.0.1 as well.
- Lines 2–3: Hand out only the virtual address.
vyos@R2:~$ show configuration commands | match "dns forwarding"
set service dns forwarding allow-from '10.0.0.0/24'
set service dns forwarding listen-address '10.0.0.3'
set service dns forwarding listen-address '10.0.0.1'
set service dns forwarding name-server 192.0.2.53
vyos@R1:~$ sudo ss -lunp | grep -E ':53 '
UNCONN 0 0 10.0.0.1:53 0.0.0.0:* users:(("pdns_recursor",pid=3812,fd=5))
UNCONN 0 0 10.0.0.2:53 0.0.0.0:* users:(("pdns_recursor",pid=3812,fd=4))- Line 4: R2 listens on the virtual address too.
- Line 7: R1, still without VRRP, already listening on 10.0.0.1.
vyos@R1:~$ show vrrp
Name Interface VRID State Priority Last Transition
------ ----------- ------ ------- ---------- -----------------
LAN eth0 10 MASTER 200 13s
WAN eth1 20 MASTER 200 13s
vyos@R2:~$ show vrrp
Name Interface VRID State Priority Last Transition
------ ----------- ------ ------- ---------- -----------------
LAN eth0 10 BACKUP 100 15s
WAN eth1 20 BACKUP 100 15s
root@pc1:~# ping -D -i 0.2 -c 200 -W 1 192.0.2.80
[... 202 lines trimmed: the header and 200 replies, none missing ...]
--- 192.0.2.80 ping statistics ---
200 packets transmitted, 200 received, 0% packet loss, time 41167ms
rtt min/avg/max/mdev = 0.279/1.041/3.023/0.727 ms- Lines 4–5: R1 takes both groups back.
- Line 14: A planned handover: nothing lost.
root@pc1:~# dhclient -r eth0; dhclient eth0; cat /etc/resolv.conf
Killed old client process
domain office.example
search office.example
nameserver 10.0.0.1
root@lab:~# date +"killing r1 at %T.%N"; pkill -f console-r1.sock
killing r1 at 14:07:39.667101677
vyos@R2:~$ show vrrp
Name Interface VRID State Priority Last Transition
------ ----------- ------ ------- ---------- -----------------
LAN eth0 10 MASTER 100 4s
WAN eth1 20 MASTER 100 4s
root@pc1:~# date +%T.%N; dig server.example.net +noall +answer +stats | grep -E "IN|SERVER"; date +%T.%N
14:07:47.595449180
server.example.net. 1 IN A 192.0.2.80
;; SERVER: 10.0.0.1#53(10.0.0.1) (UDP)
14:07:47.624149972
root@pc1:~# curl -sS -m 5 http://server.example.net/
hello from server.example.net- Line 5: pc1 now uses the virtual address for DNS.
- Line 7: R1 killed again.
- Lines 14–17: A lookup right after the failover…
- Line 16: …answered at 10.0.0.1, now by R2, in 29 ms.
- Line 19: The web works at once.
Check your understanding
After a router failover, ping works but users say "nothing loads". What's the likely cause?
- DHCP handed out the routers' own addresses as DNS servers, and clients wait for the dead one first
- VRRP failed over too slowly
- The DHCP leases expired
- The web server is down