Build the lab yourself
The lab runs on any machine with Docker, including an Apple Silicon Mac. The two routers are Debian 13 with systemd, so keepalived, conntrackd, nftables and networkd behave as on real hardware; the ISP, the web server and the staff PC are plain Debian containers.
- Put the files from this page in one directory, with each router's files in
r1/andr2/, andstream.py(the long-lived TCP stream) next tolab-setup.sh. docker compose -f lab-compose.yaml up -d --build, then./lab-setup.sh. It addresses the ISP, web server and staff PC, starts the stream server, and leaves forwarding off on the routers, as on a fresh install.- For each router, copy its
.networkand.netdevfiles to/etc/systemd/network/,99-router.confto/etc/sysctl.d/andnftables.confto/etc/, thensystemctl enable --now systemd-networkd nftablesandsysctl --system. - Copy
keepalived.confandconntrackd.confto/root/on each router, and follow the lesson.
To watch a failover, start the stream on the staff PC with timestamps: nc 198.51.100.10 9000 | while IFS= read -r l; do echo "$(date +%T.%3N) $l"; done, and kill a router from the host with docker compose -f lab-compose.yaml kill r1. Bring it back with docker compose -f lab-compose.yaml start r1.
The routers have fixed MAC addresses in lab-compose.yaml, so a restarted container is like a rebooted machine, not a new one.
# Two Debian routers: the lab. docker compose -f lab-compose.yaml up -d --build && ./lab-setup.sh
name: ha-lab
x-router: &router
build: { context: ., dockerfile: Dockerfile.router }
image: ttr-ha-router:13
privileged: true
cgroup: host # let systemd run as PID 1
volumes: ["/sys/fs/cgroup:/sys/fs/cgroup:rw"]
tmpfs: ["/run", "/run/lock"]
x-host: &host
build: { context: ., dockerfile: Dockerfile.host }
image: ttr-ha-host:13
privileged: true
services:
r1: { <<: *router, hostname: r1, networks: { wan: { interface_name: eth0, priority: 100, mac_address: "02:00:00:00:01:00" }, trunk: { interface_name: eth1, priority: 90, mac_address: "02:00:00:00:01:01" }, sync: { interface_name: eth2, mac_address: "02:00:00:00:01:02" } } }
r2: { <<: *router, hostname: r2, networks: { wan: { interface_name: eth0, priority: 100, mac_address: "02:00:00:00:02:00" }, trunk: { interface_name: eth1, priority: 90, mac_address: "02:00:00:00:02:01" }, sync: { interface_name: eth2, mac_address: "02:00:00:00:02:02" } } }
isp: { <<: *host, hostname: isp, networks: { wan: { interface_name: eth0, priority: 100 }, internet: { interface_name: eth1 } } }
web: { <<: *host, hostname: web, networks: { internet: { interface_name: eth0 } } }
staff: { <<: *host, hostname: staff, networks: { trunk: { interface_name: eth0 } } }
# Layer-2-only networks. "trunk" is the office switch (staff tags VLAN 10 itself);
# "sync" is a cable between the two routers, for conntrackd.
networks:
wan: { enable_ipv4: false, enable_ipv6: true, ipam: { config: [{ subnet: "fd00:d1::/64" }] } }
trunk: { enable_ipv4: false, enable_ipv6: true, ipam: { config: [{ subnet: "fd00:d2::/64" }] } }
sync: { enable_ipv4: false, enable_ipv6: true, ipam: { config: [{ subnet: "fd00:d3::/64" }] } }
internet: { enable_ipv4: false, enable_ipv6: true, ipam: { config: [{ subnet: "fd00:d4::/64" }] } }#!/bin/sh
# Address everything except the routers (their configuration is the lesson). Run after `docker compose up`.
x() { docker compose -f lab-compose.yaml exec -T "$1" sh -c "$2"; }
for n in r1 r2 isp web staff; do x $n 'sysctl -qw net.ipv6.conf.all.disable_ipv6=1'; done
# Docker Desktop turns forwarding on in every container; a fresh Debian install has it off.
for n in r1 r2; do x $n 'sysctl -qw net.ipv4.ip_forward=0'; done
x isp 'ip addr add 203.0.113.1/24 dev eth0; ip addr add 198.51.100.1/24 dev eth1; sysctl -qw net.ipv4.ip_forward=1'
x web 'ip addr add 198.51.100.10/24 dev eth0; ip route add default via 198.51.100.1'
docker compose -f lab-compose.yaml cp stream.py web:/usr/local/bin/stream.py >/dev/null 2>&1
x web 'nohup python3 /usr/local/bin/stream.py > /dev/null 2>&1 &'
# A staff PC on VLAN 10, whose gateway is the routers' shared address, 10.10.10.1.
x staff 'ip link add link eth0 name eth0.10 type vlan id 10 && ip link set eth0.10 up && ip addr add 10.10.10.20/24 dev eth0.10 && ip route add default via 10.10.10.1'#!/usr/bin/env python3
# A long-lived TCP connection to watch during a failover: every 0.2 s, send a counter and the time.
import socketserver, time
class Stream(socketserver.BaseRequestHandler):
def handle(self):
n = 0
while True:
n += 1
self.request.sendall(f"{n} {time.strftime('%H:%M:%S')}\n".encode())
time.sleep(0.2)
socketserver.ThreadingTCPServer.allow_reuse_address = True
socketserver.ThreadingTCPServer(("", 9000), Stream).serve_forever()# The two routers: Debian 13 with systemd, udev and D-Bus (as in "Debian as an office router"),
# plus keepalived for VRRP and conntrackd to share connection state.
FROM debian:trixie-slim
RUN apt-get update && apt-get install -y --no-install-recommends \
systemd systemd-sysv udev dbus nftables keepalived conntrackd conntrack netbase iproute2 iputils-ping \
tcpdump procps less \
&& rm -rf /var/lib/apt/lists/* \
&& systemctl mask systemd-modules-load.service getty@.service autovt@.service getty-static.service
CMD ["/sbin/init"]- Line 5: keepalived and conntrackd, and netbase for /etc/protocols, which conntrackd's filter needs.
# Office PCs, servers and the internet: plain Debian 13 with network tools and a DHCP client.
FROM debian:trixie-slim
RUN apt-get update && apt-get install -y --no-install-recommends \
iproute2 iputils-ping tcpdump curl python3 netcat-openbsd procps dnsutils dhcpcd-base \
&& rm -rf /var/lib/apt/lists/*
CMD ["sleep", "infinity"]# The internet side: this router's own address. The shared address, 203.0.113.4, is keepalived's.
[Match]
Name=eth0
[Network]
Address=203.0.113.2/24
Gateway=203.0.113.1# The office side: the trunk to the switch, carrying VLAN 10 (staff).
[Match]
Name=eth1
[Network]
VLAN=staff
LinkLocalAddressing=no# VLAN 10: staff PCs. Their gateway is the shared address, 10.10.10.1.
[NetDev]
Name=staff
Kind=vlan
[VLAN]
Id=10# This router's own address on VLAN 10; keepalived adds 10.10.10.1 on the active router.
[Match]
Name=staff
[Network]
Address=10.10.10.2/24# A direct cable to the other router, for conntrackd.
[Match]
Name=eth2
[Network]
Address=10.99.0.1/30
LinkLocalAddressing=no# The internet side: this router's own address. The shared address, 203.0.113.4, is keepalived's.
[Match]
Name=eth0
[Network]
Address=203.0.113.3/24
Gateway=203.0.113.1# This router's own address on VLAN 10; keepalived adds 10.10.10.1 on the active router.
[Match]
Name=staff
[Network]
Address=10.10.10.3/24# A direct cable to the other router, for conntrackd.
[Match]
Name=eth2
[Network]
Address=10.99.0.2/30
LinkLocalAddressing=no# /etc/keepalived/keepalived.conf on r2
global_defs {
router_id r2
enable_script_security
script_user root
}
# Is the ISP reachable from here? Two failed pings in a row and this router steps down.
vrrp_script isp_reachable {
script "/usr/bin/ping -c 1 -W 1 203.0.113.1"
interval 2
fall 2
rise 2
}
# Both addresses move together: a router that holds one must hold the other.
vrrp_sync_group OFFICE {
group {
STAFF
WAN
}
}
vrrp_instance STAFF {
state BACKUP
interface staff
virtual_router_id 10
priority 100
advert_int 1
virtual_ipaddress {
10.10.10.1/24
}
track_script {
isp_reachable
}
}
vrrp_instance WAN {
state BACKUP
interface eth0
virtual_router_id 20
priority 100
advert_int 1
virtual_ipaddress {
203.0.113.4/24
}
}# /etc/conntrackd/conntrackd.conf on r2: send our connection table to the other router, and
# put theirs straight into our kernel, so either router can carry on any connection.
Sync {
Mode FTFW {
DisableExternalCache On
}
UDP {
IPv4_address 10.99.0.2
IPv4_Destination_Address 10.99.0.1
Port 3780
Interface eth2
Checksum on
}
}
General {
HashSize 8192
HashLimit 65535
Syslog on
LockFile /var/lock/conntrackd.lock
UNIX {
Path /var/run/conntrackd.sock
}
Systemd on
Filter From Userspace {
Protocol Accept {
TCP
UDP
ICMP
}
Address Ignore {
IPv4_address 127.0.0.1
IPv4_address 10.99.0.0/30
}
}
}# /etc/sysctl.d/99-router.conf: route between the WAN and the VLANs.
net.ipv4.ip_forward = 1Where next
- Debian as an office router: interfaces, VLANs and a firewallPrerequisite · Intermediate · 45 min
Turn a fresh Debian 13 install into an office router: systemd-networkd for the WAN and three VLANs on one trunk, IP forwarding, an nftables policy between VLANs with NAT, protecting the router itself, and a configuration that survives a reboot.
- Office gateway: DHCP, DNS and VRRP on VyOSAlso covers VRRP, high availability · Beginner · 40 min
Two VyOS routers serving one office LAN: VRRP for a gateway that survives a router failure, DHCP with split pools and a printer reservation, and DNS forwarding. Then a real failover, the DNS outage it exposed, and the fix.
- nftables and iptables on Debian: host and router firewallsAlso covers debian, nftables · Intermediate · 60 min
Build a Debian firewall with nftables, from a host firewall to an office router with NAT and port forwards: safe changes with a timed rollback, sets and maps, logging and packet tracing, and what iptables really is on Debian 13.
- OpenVPN on Debian: users, routing and revocationAlso covers debian, nftables · Intermediate · 60 min
Install and run an OpenVPN server on Debian with your own certificate authority: one certificate per user, fixed addresses, routing to an office and a branch LAN, split and full tunnels, tls-crypt, revoking users, and the mistakes that lock everyone out.