Intermediate · Packets That Never Come Back · section 3 of 7

The VyOS lab

✓ Lab-tested 17 October 2026 on VyOS 2026.10.16-0024-rolling, Debian 12

VyOS is a Debian-based router OS. Its data plane is the Linux kernel (routing, nftables firewall and NAT, conntrack), with FRR for dynamic routing. Configuration is a single tree edited with set/delete in configuration mode and applied atomically with commit.

Three VyOS commands do most of the troubleshooting:

monitor traffic interface ethX filter "…" is tcpdump on the router, with the same BPF filters. Add save /tmp/x.pcap to write a file instead.
show ip route, show nat source translations and show firewall are the router's view of why.
sudo …: you can always drop to the underlying Linux tools (nft, conntrack, ip) when a summary isn't enough.

Running real VyOS anywhere

VyOS publishes rolling builds as an x86-64 ISO. This lab runs it as a full virtual machine under QEMU inside a container. Each VM NIC is bridged to a Docker network, so VyOS sits on the same layer-2 segments as the Linux hosts. That works on any Docker host, including ARM Macs, where the x86 VM is emulated (slow to boot, fine for labs).

A live-ISO boot keeps configuration in memory only, hence the live-ISO environment and not been saved warnings in the transcripts.

Get the ISO

# Download the VyOS rolling ISO used in this lesson and extract its kernel and initrd.
mkdir -p vyos && cd vyos
curl -LO https://github.com/vyos/vyos-nightly-build/releases/download/2026.10.16-0024-rolling/vyos-2026.10.16-0024-rolling-generic-amd64.iso
mv vyos-*-amd64.iso vyos.iso
tar -xf vyos.iso live/vmlinuz live/initrd.img   # bsdtar (macOS) reads ISOs; on Linux use 7z or bsdtar
mv live/vmlinuz live/initrd.img . && rmdir live

Line 3: The exact rolling build used for this lesson. Old rolling builds are eventually removed; any recent one from the vyos-nightly-build releases page will work.
Lines 5–6: QEMU boots the kernel and initrd directly, so we pull them out of the ISO. The ISO itself is still attached as a CD: it holds the root filesystem.

start-vyos.sh: a VM inside a container

#!/bin/sh
# Bridge each container interface to a tap device, then boot VyOS with one virtio NIC per tap.
set -e
n=0
for ifc in eth0 eth1; do
  ip link add "br$n" type bridge
  ip link set "$ifc" master "br$n"
  ip tuntap add "tap$n" mode tap
  ip link set "tap$n" master "br$n"
  ip link set "tap$n" up
  ip link set "br$n" up
  # Pin the VM's MAC to its tap and stop learning on the Docker side: Docker reflects the VM's
  # broadcasts back, which would otherwise teach the bridge the VM's MAC on the wrong port.
  bridge link set dev "$ifc" learning off
  bridge fdb replace "52:54:00:00:0$n:01" dev "tap$n" master static
  n=$((n + 1))
done
exec qemu-system-x86_64 -m 2048 -smp 2 -cpu max -accel tcg,thread=multi \
  -kernel /vyos/vmlinuz -initrd /vyos/initrd.img \
  -append "boot=live components hostname=vyos username=live nopersistence noautologin nonetworking union=overlay console=ttyS0,115200 net.ifnames=0 biosdevname=0" \
  -cdrom /vyos/vyos.iso \
  -netdev tap,id=n0,ifname=tap0,script=no,downscript=no -device virtio-net-pci,netdev=n0,mac=52:54:00:00:00:01 \
  -netdev tap,id=n1,ifname=tap1,script=no,downscript=no -device virtio-net-pci,netdev=n1,mac=52:54:00:00:01:01 \
  -display none -serial unix:/run/console.sock,server=on,wait=off -monitor none

Lines 5–17: For each container interface (eth0 = LAN, eth1 = WAN): create a bridge and a tap device and join them, so the VM's NIC lands on that Docker network.
Lines 12–15: Pin each VM MAC to its tap port and turn off MAC learning on the Docker side. Without this, Docker reflects the VM's broadcasts back, the bridge learns the VM's MAC on the wrong port, and replies to the VM are intermittently lost.
Line 18: Software emulation (TCG). On an x86 Linux host you can add -enable-kvm and it becomes fast.
Line 20: The ISO's own "serial console" boot parameters, so the console is a socket we can attach to.
Lines 22–23: One virtio NIC per tap, with fixed MACs: 52:54:00:00:00:01 becomes VyOS eth0, 52:54:00:00:01:01 eth1.
Line 24: No screen. The serial console is exposed as a unix socket.

Lab compose file

# Run get-vyos.sh first, then: docker compose up -d --build
# VyOS console: docker compose exec -it vyos socat -,raw,echo=0,escape=0x1d UNIX-CONNECT:/run/console.sock
#   (log in as vyos / vyos; Ctrl-] to leave the console)
name: vyos-lab
x-linux: &linux
  image: ttr-linux-lab
  privileged: true
services:
  vyos:
    build:
      context: .
      dockerfile_inline: |
        FROM debian:bookworm-slim
        RUN apt-get update && apt-get install -y --no-install-recommends qemu-system-x86 iproute2 socat
        COPY start-vyos.sh /usr/local/bin/start-vyos.sh
        CMD ["/usr/local/bin/start-vyos.sh"]
    cap_add: [NET_ADMIN]
    devices: ["/dev/net/tun:/dev/net/tun"]
    volumes: ["./vyos:/vyos:ro"]
    networks: { lan: { interface_name: eth0 }, wan: { interface_name: eth1 } }
  client:
    <<: *linux
    build:
      dockerfile_inline: |
        FROM debian:bookworm-slim
        RUN apt-get update && apt-get install -y --no-install-recommends \
          iproute2 iputils-ping traceroute dnsutils curl tcpdump netcat-openbsd \
          nftables python3 procps ethtool
        CMD ["sleep", "infinity"]
    hostname: client
    networks: { lan: { interface_name: eth0 } }
  rtr2:
    <<: *linux
    hostname: rtr2
    networks: { lan: { interface_name: eth0 }, wan: { interface_name: eth1 } }
  server:
    <<: *linux
    hostname: server
    networks: { wan: { interface_name: eth0 } }
networks:
  lan: { enable_ipv4: false, enable_ipv6: true, ipam: { config: [{ subnet: "fd00:c::/64" }] } }
  wan: { enable_ipv4: false, enable_ipv6: true, ipam: { config: [{ subnet: "fd00:d::/64" }] } }

Lines 2–3: Attach to the VyOS serial console with socat. Ctrl-] detaches without stopping the VM.
Lines 17–18: The VM container needs NET_ADMIN and /dev/net/tun to create bridges and taps. It does not need to be privileged.
Lines 40–42: Layer-2-only networks again: every IPv4 address comes from the VyOS config and hosts-setup.sh.

Linux hosts

# client: on the LAN, default gateway is VyOS. Offloads off so tcpdump shows real segment sizes.
ip addr add 10.0.0.10/24 dev eth0
ip route add default via 10.0.0.1
ethtool -K eth0 tso off gso off gro off

# rtr2: a second router between the same two networks (used to create asymmetry)
ip addr add 10.0.0.2/24 dev eth0
ip addr add 203.0.113.2/24 dev eth1
sysctl -w net.ipv4.ip_forward=1

# server: on the WAN, with no route back to 10.0.0.0/24 (like a host on the internet)
ip addr add 203.0.113.20/24 dev eth0
ethtool -K eth0 tso off gso off gro off
cd /srv && head -c 300000 /dev/urandom > big.bin && echo hello > index.html
python3 -m http.server 80 &

Line 4: Segmentation offloads off, so tcpdump shows the packets that actually go on the wire instead of 64 KB super-packets.
Line 9: rtr2 is a plain forwarding router with no firewall. It exists only to make paths asymmetric.
Lines 11–12: Deliberately no default route: the server behaves like a host on the internet that knows nothing about 10.0.0.0/24.

R1 base configuration (typed on the VyOS console)

configure
set system host-name R1
set interfaces ethernet eth0 address 10.0.0.1/24
set interfaces ethernet eth0 description LAN
set interfaces ethernet eth1 address 203.0.113.1/24
set interfaces ethernet eth1 description WAN
commit

Line 1: Enter configuration mode. The prompt changes from $ to #.
Lines 3–6: Addresses and descriptions on both interfaces. Nothing is active yet…
Line 7: …until commit applies the whole change atomically. (save would persist it, which a live-ISO boot can't.)