TTR Networks

Intermediate · Two internet connections on VyOS · section 7 of 7

Build the lab yourself

✓ Lab-tested 21 October 2026 on VyOS 2026.10.16-0024-rolling, Debian 12 hosts

The lab is one VyOS VM and four small Debian containers. It needs about 1.5 GB of free memory for Docker.

  1. Get the VyOS image as in the tcpdump lesson (get-vyos.sh), into ./vyos.
  2. Put lab-compose.yaml, start-vyos.sh, hosts-setup.sh, failover-test.sh, listen.sh and load.sh in one directory. Then docker compose up -d --build and ./hosts-setup.sh.
  3. Copy the two helper scripts into the hosts: docker compose cp listen.sh server:/usr/local/bin/ and docker compose cp load.sh lanhost:/usr/local/bin/.
  4. When VyOS has booted, paste the stages in order from its console.
  5. Break ISP1's upstream with docker compose exec isp1 ip link set eth1 down, and bring it back with up. When you bring it back, also run ip route replace 198.51.100.0/30 via 192.0.2.2 on isp1: taking a link down removes its routes, as on any Linux router. ./failover-test.sh NAME does all of this and records the results.
lab-compose.yaml
# Needs ./vyos (from the tcpdump-and-vyos lesson's get-vyos.sh). Then: docker compose up -d --build && ./hosts-setup.sh
# VyOS console: docker compose exec -it r1 socat -,raw,echo=0,escape=0x1d UNIX-CONNECT:/run/console.sock
name: vdwan-lab
x-host: &host
  build:
    dockerfile_inline: |
      FROM debian:bookworm-slim
      RUN apt-get update && apt-get install -y --no-install-recommends \
        iproute2 iputils-ping curl tcpdump netcat-openbsd python3 procps
      CMD ["sleep", "infinity"]
  image: ttr-vdwan-host
  privileged: true
services:
  r1:
    build:
      context: .
      dockerfile_inline: |
        FROM debian:bookworm-slim
        RUN apt-get update && apt-get install -y --no-install-recommends qemu-system-x86 iproute2 socat
        COPY start-vyos.sh /usr/local/bin/start-vyos.sh
        CMD ["/usr/local/bin/start-vyos.sh"]
    cap_add: [NET_ADMIN]
    devices: ["/dev/net/tun:/dev/net/tun"]
    environment: { VYOS_MEM: "1536", VYOS_SMP: "1", VM_ID: "31" }
    volumes: ["./vyos:/vyos:ro"]
    networks: { wan1: { interface_name: eth0 }, wan2: { interface_name: eth1 }, lan: { interface_name: eth2 } }
  isp1:    { <<: *host, hostname: isp1,    networks: { wan1: { interface_name: eth0, priority: 100 }, internet: { interface_name: eth1 } } }
  isp2:    { <<: *host, hostname: isp2,    networks: { wan2: { interface_name: eth0, priority: 100 }, internet: { interface_name: eth1 } } }
  server:  { <<: *host, hostname: server,  networks: { internet: { interface_name: eth0 } } }
  lanhost: { <<: *host, hostname: lanhost, networks: { lan: { interface_name: eth0 } } }
# Layer-2-only networks: every address comes from VyOS's config and hosts-setup.sh.
networks:
  wan1:     { enable_ipv4: false, enable_ipv6: true, ipam: { config: [{ subnet: "fd00:b1::/64" }] } }
  wan2:     { enable_ipv4: false, enable_ipv6: true, ipam: { config: [{ subnet: "fd00:b2::/64" }] } }
  lan:      { enable_ipv4: false, enable_ipv6: true, ipam: { config: [{ subnet: "fd00:b3::/64" }] } }
  internet: { enable_ipv4: false, enable_ipv6: true, ipam: { config: [{ subnet: "fd00:b4::/64" }] } }
listen.sh
#!/bin/sh
# On the server: accept one long TCP connection on port 9000 and stamp each line with its arrival time.
nc -l 9000 | while read -r l; do echo "$l, received $(date +%T)"; done
start-vyos.sh
#!/bin/sh
# Bridge every container interface (eth0, eth1, …) to a tap, then boot VyOS with one virtio NIC per tap.
# VM NIC n gets MAC 52:54:00:<VM_ID>:0n:01, so VyOS's ethN matches the container's ethN. Give each VM its own VM_ID.
set -e
NETDEVS=""
n=0
for ifc in $(ls /sys/class/net | grep -E '^eth[0-9]+$' | sort -V); do
  ip link add "br$n" type bridge
  ip link set "$ifc" master "br$n"
  ip tuntap add "tap$n" mode tap
  ip link set "tap$n" master "br$n"
  ip link set "tap$n" up
  ip link set "br$n" up
  # Pin the VM's MAC to its tap and stop learning on the Docker side: Docker reflects the VM's
  # broadcasts back, which would otherwise teach the bridge the VM's MAC on the wrong port.
  bridge link set dev "$ifc" learning off
  bridge fdb replace "52:54:00:${VM_ID:-00}:0$n:01" dev "tap$n" master static
  NETDEVS="$NETDEVS -netdev tap,id=n$n,ifname=tap$n,script=no,downscript=no -device virtio-net-pci,netdev=n$n,mac=52:54:00:${VM_ID:-00}:0$n:01"
  n=$((n + 1))
done
# VYOS_TB_MB caps QEMU's TCG translation cache (QEMU's default is 1 GB on top of the guest RAM).
exec qemu-system-x86_64 -m "${VYOS_MEM:-2048}" -smp "${VYOS_SMP:-2}" -cpu max -accel tcg,thread=multi,tb-size=${VYOS_TB_MB:-256} \
  -kernel /vyos/vmlinuz -initrd /vyos/initrd.img \
  -append "boot=live components hostname=vyos username=live nopersistence noautologin nonetworking union=overlay console=ttyS0,115200 net.ifnames=0 biosdevname=0" \
  -cdrom /vyos/vyos.iso $NETDEVS \
  -display none -serial unix:/run/console.sock,server=on,wait=off -monitor none

Where next

  • Setting up VyOS with NATPrerequisite · Beginner · 45 min

    A VyOS router with a NAT interface for a private LAN and a public interface for a routed public subnet: plain routing, source NAT, port forwarding and the forward firewall that NAT isn't. Every output comes from a real VyOS lab.

  • Two internet connections on DebianAlso covers failover, policy routing · Intermediate · 45 min

    Give a Debian office router a second ISP: a routing table per ISP, source rules and connection marks so replies leave the way they came, per-ISP NAT, and a systemd health check that fails over and back, measured with a real outage.

  • Zone-based firewall on VyOSAlso covers NAT · Intermediate · 35 min

    Rebuild the NAT lesson's router with a zone-based firewall: WAN, LAN, a DMZ and the router itself, with one ruleset per direction. Includes the missing-return-traffic mistake, post-NAT matching, and a full test matrix.

  • nftables and iptables on Debian: host and router firewallsAlso covers NAT · Intermediate · 60 min

    Build a Debian firewall with nftables, from a host firewall to an office router with NAT and port forwards: safe changes with a timed rollback, sets and maps, logging and packet tracing, and what iptables really is on Debian 13.