TTR Networks

Intermediate · Traffic shaping with CAKE on Debian · section 8 of 8

Build the lab yourself

✓ Lab-tested 5 November 2026 on Debian 13.7 (kernel 6.12, in a QEMU VM), iproute2 6.15.0, systemd 257, iperf3 3.18

The router is a Debian 13 virtual machine running under QEMU in a container; the PCs, the modem and the server are containers. It runs on any machine with Docker, including an Apple Silicon Mac (QEMU emulates the x86 VM), with about 2 GB of free memory.

  1. Put the files from this page in one directory. Put user-data, meta-data and network-config in a subdirectory called seed: cloud-init reads them on the VM's first boot, to set its hostname and the lab's root password.
  2. Run ./get-debian-image.sh. It downloads Debian's official generic cloud image (about 420 MB) into img/ and checks it against Debian's published checksums. The VM boots from a copy-on-write layer on top of it, so the download stays untouched.
  3. docker compose -f lab-compose.yaml up -d --build, then ./lab-setup.sh to address the containers and make the modem's slow, deep-buffered line (Linux's tbf qdisc).
  4. The VM takes a few minutes to boot under emulation. Drive its serial console with vmcon.py, which the compose file mounts into the QEMU container: docker compose -f lab-compose.yaml exec gw python3 /work/vmcon.py wait, then … login, then … run 'ip -br addr'. The root password is lab.
  5. On the VM, add the two .network files (start 20-wan.network without its [CAKE] section) and 99-forwarding.conf, then networkctl reload and sysctl --system. Measure first, then follow the lesson.

The modem's buffer and speeds are set in lab-setup.sh: change them to match your own line and see how CAKE's settings need to follow.

lab-compose.yaml
# Traffic shaping with CAKE: the lab.  docker compose -f lab-compose.yaml up -d --build && ./lab-setup.sh
# The router is a real Debian VM (Docker Desktop's kernel has no CAKE); everything else is a container.
name: qos-lab
x-host: &host
  build: { context: ., dockerfile: Dockerfile.host }
  image: ttr-qos-host:13
  privileged: true
services:
  gw:
    build: { context: ., dockerfile: Dockerfile.qemu }
    image: ttr-qos-qemu:13
    hostname: gw-vm
    privileged: true
    volumes: ["./img:/img", "./seed:/seed:ro", "./vmcon.py:/work/vmcon.py:ro"]
    networks: { lan: { interface_name: eth0, priority: 100 }, wan: { interface_name: eth1 } }
  client:  { <<: *host, hostname: client,  networks: { lan: { interface_name: eth0 } } }
  client2: { <<: *host, hostname: client2, networks: { lan: { interface_name: eth0 } } }
  modem:   { <<: *host, hostname: modem,   networks: { wan: { interface_name: eth0, priority: 100 }, net: { interface_name: eth1 } } }
  server:  { <<: *host, hostname: server,  networks: { net: { interface_name: eth0 } } }
networks:
  lan: { enable_ipv4: false, enable_ipv6: true, ipam: { config: [{ subnet: "fd00:d6::/64" }] } }
  wan: { enable_ipv4: false, enable_ipv6: true, ipam: { config: [{ subnet: "fd00:d7::/64" }] } }
  net: { enable_ipv4: false, enable_ipv6: true, ipam: { config: [{ subnet: "fd00:d8::/64" }] } }
lab-setup.sh
#!/bin/sh
# Address the containers and make the modem's slow line. Run after `docker compose up`.
# The router VM (gw) is configured from its console, as in the lesson.
x() { docker compose -f lab-compose.yaml exec -T "$1" sh -c "$2"; }
for n in client client2 modem server; do x $n 'sysctl -qw net.ipv6.conf.all.disable_ipv6=1'; done
x client  'ip addr add 10.10.0.10/24 dev eth0; ip route add default via 10.10.0.1'
x client2 'ip addr add 10.10.0.11/24 dev eth0; ip route add default via 10.10.0.1'
x server  'ip addr add 198.51.100.10/24 dev eth0; ip route add default via 198.51.100.1; iperf3 -s -D'
x modem   'ip addr add 192.0.2.1/24 dev eth0; ip addr add 198.51.100.1/24 dev eth1; ip route add 10.10.0.0/24 via 192.0.2.2; sysctl -qw net.ipv4.ip_forward=1'
# The "broadband line": 5 Mbit/s up and 20 Mbit/s down, each with a deep buffer (up to a second),
# like many real modems. The queue that hurts builds up here, not in the router.
x modem   'tc qdisc add dev eth1 root tbf rate 5mbit burst 16kb latency 1000ms'
x modem   'tc qdisc add dev eth0 root tbf rate 20mbit burst 32kb latency 500ms'
  • Lines 10–13: The modem's line: tbf at 5 and 20 Mbit/s, with deep buffers.
get-debian-image.sh
#!/bin/sh
# Download Debian 13's official "generic" cloud image into ./img and check it against Debian's SHA512SUMS.
set -e
base=https://cloud.debian.org/images/cloud/trixie/latest
img=debian-13-generic-amd64.qcow2
mkdir -p img
curl -fL -o "img/$img" "$base/$img"
want=$(curl -fsSL "$base/SHA512SUMS" | awk -v f="$img" '$2 == f {print $1}')
got=$( (sha512sum "img/$img" 2>/dev/null || shasum -a 512 "img/$img") | awk '{print $1}')
[ "$want" = "$got" ] && echo "checksum OK" || { echo "checksum mismatch" >&2; exit 1; }
Dockerfile.qemu
# Runs one Debian 13 VM (its own kernel, with CAKE) as the office router.
FROM debian:trixie-slim
RUN apt-get update && apt-get install -y --no-install-recommends \
      qemu-system-x86 qemu-utils cloud-image-utils iproute2 python3 procps \
 && rm -rf /var/lib/apt/lists/*
COPY start-gw.sh /usr/local/bin/start-gw.sh
CMD ["/usr/local/bin/start-gw.sh"]
start-gw.sh
#!/bin/sh
# Bridge each container interface to a tap, then boot the Debian VM with one virtio NIC per tap.
set -e
n=0
for ifc in eth0 eth1; do
  ip link add "br$n" type bridge
  ip link set "$ifc" master "br$n"
  ip tuntap add "tap$n" mode tap
  ip link set "tap$n" master "br$n"
  ip link set "tap$n" up
  ip link set "br$n" up
  bridge link set dev "$ifc" learning off
  bridge fdb replace "52:54:00:00:0$n:01" dev "tap$n" master static
  n=$((n + 1))
done
# A copy-on-write disk on top of Debian's image, so the download stays untouched.
[ -f /img/gw.qcow2 ] || qemu-img create -q -f qcow2 -b debian-13-generic-amd64.qcow2 -F qcow2 /img/gw.qcow2 8G
# cloud-init's first-boot settings (a hostname and a root password) on a small "cidata" disk.
[ -f /img/seed.iso ] || cloud-localds -N /seed/network-config /img/seed.iso /seed/user-data /seed/meta-data
exec qemu-system-x86_64 -m 1024 -smp 1 -cpu max -accel tcg,tb-size=256 \
  -drive file=/img/gw.qcow2,if=virtio -drive file=/img/seed.iso,format=raw,if=virtio \
  -netdev tap,id=n0,ifname=tap0,script=no,downscript=no -device virtio-net-pci,netdev=n0,mac=52:54:00:00:00:01 \
  -netdev tap,id=n1,ifname=tap1,script=no,downscript=no -device virtio-net-pci,netdev=n1,mac=52:54:00:00:01:01 \
  -display none -serial unix:/run/console.sock,server=on,wait=off -monitor none
  • Lines 16–19: A copy-on-write disk over Debian's image, and cloud-init's seed disk.
  • Lines 20–24: One vCPU, a smaller translation cache, two virtio NICs bridged to the Docker networks, and a serial console on a socket.
Dockerfile.host
# The LAN client, the "modem" and the internet server: Debian 13 with network tools.
FROM debian:trixie-slim
RUN apt-get update && apt-get install -y --no-install-recommends \
      iproute2 iputils-ping iperf3 procps \
 && rm -rf /var/lib/apt/lists/*
CMD ["sleep", "infinity"]
seed/user-data (cloud-init)
#cloud-config
# First boot of the lab's router VM: a name, and a root password for the serial console.
hostname: gw
chpasswd:
  expire: false
  users:
    - { name: root, password: lab, type: text }
disable_root: false
seed/meta-data
instance-id: gw
local-hostname: gw
seed/network-config
network:
  config: disabled
/etc/systemd/network/10-lan.network
# /etc/systemd/network/10-lan.network: the office LAN.
[Match]
Name=ens3

[Network]
Address=10.10.0.1/24
LinkLocalAddressing=no
IPv6AcceptRA=no
/etc/sysctl.d/99-forwarding.conf
# /etc/sysctl.d/99-forwarding.conf: this machine is a router.
net.ipv4.ip_forward = 1
vmcon.py (drives the VM's serial console)
#!/usr/bin/env python3
"""Drive the Debian VM's serial console (QEMU unix socket).

  vmcon.py wait [secs]          wait for the login prompt
  vmcon.py login                log in as root (password set by cloud-init from seed/user-data)
  vmcon.py run 'cmd' [secs]     run one shell command, print its output (no prompt, no echo)
"""
import os, re, socket, sys, time

SOCK = '/run/console.sock'
ANSI = re.compile(r'\x1b\[[0-9;?]*[ -/]*[@-~]|\x1b[=>()][0-9A-Za-z]?|\r|\x07|\x08')

def connect():
    s = socket.socket(socket.AF_UNIX); s.connect(SOCK); s.settimeout(0.3); return s

def read_until(s, pattern, timeout):
    buf, end = '', time.time() + timeout
    while time.time() < end:
        try: data = s.recv(65536)
        except socket.timeout: data = b''
        if data:
            buf += data.decode('utf-8', 'replace')
            if re.search(pattern, ANSI.sub('', buf)): return ANSI.sub('', buf), True
    return ANSI.sub('', buf), False

cmd = sys.argv[1]
s = connect()
if cmd == 'wait':
    t = float(sys.argv[2]) if len(sys.argv) > 2 else 600
    end = time.time() + t
    while time.time() < end:
        s.sendall(b'\n')
        out, ok = read_until(s, r'login: $|# $', 5)
        if ok: print('ready'); sys.exit(0)
    sys.exit('timed out')
elif cmd == 'login':
    s.sendall(b'\n'); out, _ = read_until(s, r'login: $|# $', 10)
    if out.rstrip().endswith('login:'):
        s.sendall(b'root\n'); out, ok = read_until(s, r'# $|Password: $', 30)
        if out.rstrip().endswith('Password:'):
            s.sendall(b'lab\n'); out, ok = read_until(s, r'# $', 30)  # the lab's root password, set by cloud-init
        if not ok: sys.exit('login failed: ' + out[-300:])
    # A quiet, predictable shell: no colour, no line wrapping games, no kernel messages on the console.
    s.sendall(b"stty -echo cols 250; export TERM=dumb PS1='# '; dmesg -n 1\n"); read_until(s, r'# $', 10)
    print('logged in')
elif cmd == 'run':
    t = float(sys.argv[3]) if len(sys.argv) > 3 else 120
    mark = '__DONE_%d__' % int(time.time() * 1000)
    s.sendall(b'\n'); read_until(s, r'# $', 5)
    s.sendall((sys.argv[2] + '; echo ' + mark + '\n').encode())
    out, ok = read_until(s, mark + r'\n', t)
    body = out.split(mark)[0] if ok else out
    lines = body.split('\n')
    # Drop the echoed command line (if echo is on) and any leading prompt.
    while lines and (lines[0].strip() in ('', '#') or sys.argv[2][:40] in lines[0]):
        lines.pop(0)
    print('\n'.join(l for l in lines).rstrip('\n'))
    if not ok: sys.exit('timed out')

Where next

  • Debian as an office router: interfaces, VLANs and a firewallPrerequisite · Intermediate · 45 min

    Turn a fresh Debian 13 install into an office router: systemd-networkd for the WAN and three VLANs on one trunk, IP forwarding, an nftables policy between VLANs with NAT, protecting the router itself, and a configuration that survives a reboot.

  • nftables and iptables on Debian: host and router firewallsAlso covers debian · Intermediate · 60 min

    Build a Debian firewall with nftables, from a host firewall to an office router with NAT and port forwards: safe changes with a timed rollback, sets and maps, logging and packet tracing, and what iptables really is on Debian 13.

  • OpenVPN on Debian: users, routing and revocationAlso covers debian · Intermediate · 60 min

    Install and run an OpenVPN server on Debian with your own certificate authority: one certificate per user, fixed addresses, routing to an office and a branch LAN, split and full tunnels, tls-crypt, revoking users, and the mistakes that lock everyone out.

  • WireGuard on Debian: users, routing and keysAlso covers debian · Intermediate · 50 min

    Run a WireGuard server on Debian for several users, an office LAN and a branch: keys, wg-quick and systemd, AllowedIPs as routing and access control, split and full tunnels, roaming and keepalives, adding and removing users live, and the silent failures.