TTR Networks

Intermediate · Edge and core: building out a VyOS network · section 10 of 10

Build the lab yourself

✓ Lab-tested 17 October 2026 on VyOS 2026.10.16-0024-rolling, FRR 10.2.1

Three VyOS VMs plus the same FRR upstream, route server and IX member as the previous lesson. Each VM takes about 1.5 GB of RAM, so give Docker at least 6 GB of free memory. If a VM container exits with code 137, Docker ran out of memory and killed it: lower that VM's VYOS_MEM to 1280. Save the files below in one directory, add ./vyos (from the Packets That Never Come Back lesson's get-vyos.sh) and FRR's daemons file, then docker compose up -d --build. Boot takes a few minutes under emulation. Log in to each console as vyos/vyos and paste each router's configuration blocks from this lesson in order, committing after each one.

lab-compose.yaml
# Needs ./vyos (from the tcpdump-and-vyos lesson's get-vyos.sh) and ./daemons (copy /etc/frr/daemons out of the FRR image, set bgpd=yes). Then: docker compose up -d --build
# Console for one router: docker compose exec -it core1 socat -,raw,echo=0,escape=0x1d UNIX-CONNECT:/run/console.sock
name: vyos-edge-core-lab
x-frr: &frr
  image: quay.io/frrouting/frr:10.2.1
  privileged: true
x-vyos: &vyos
  build:
    context: .
    dockerfile_inline: |
      FROM debian:bookworm-slim
      RUN apt-get update && apt-get install -y --no-install-recommends qemu-system-x86 iproute2 socat
      COPY start-vyos.sh /usr/local/bin/start-vyos.sh
      CMD ["/usr/local/bin/start-vyos.sh"]
  cap_add: [NET_ADMIN]
  devices: ["/dev/net/tun:/dev/net/tun"]
  volumes: ["./vyos:/vyos:ro"]
services:
  edge1:
    <<: *vyos
    environment: { VYOS_MEM: "1536", VM_ID: "01" }
    networks: { transit: { interface_name: eth0 }, e1-core: { interface_name: eth1 }, e1-e2: { interface_name: eth2 } }
  edge2:
    <<: *vyos
    environment: { VYOS_MEM: "1536", VM_ID: "02" }
    networks: { ix: { interface_name: eth0 }, e2-core: { interface_name: eth1 }, e1-e2: { interface_name: eth2 } }
  core1:
    <<: *vyos
    environment: { VYOS_MEM: "1536", VYOS_SMP: "1", VM_ID: "03" }
    networks: { e1-core: { interface_name: eth0 }, e2-core: { interface_name: eth1 }, lan: { interface_name: eth2 } }
  upstream:
    <<: *frr
    volumes: ["./daemons:/etc/frr/daemons:ro", "./upstream.conf:/etc/frr/frr.conf"]
    networks: { transit: { interface_name: eth0 } }
  rs1:
    <<: *frr
    volumes: ["./daemons:/etc/frr/daemons:ro", "./rs.conf:/etc/frr/frr.conf"]
    networks: { ix: { interface_name: eth0 } }
  member:
    <<: *frr
    volumes: ["./daemons:/etc/frr/daemons:ro", "./member.conf:/etc/frr/frr.conf"]
    networks: { ix: { interface_name: eth0 } }
  host:
    build:
      dockerfile_inline: |
        FROM debian:bookworm-slim
        RUN apt-get update && apt-get install -y --no-install-recommends iproute2 iputils-ping traceroute
        CMD ["sh", "-c", "ip addr add 198.51.100.10/25 dev eth0 && ip route replace default via 198.51.100.1 && exec sleep infinity"]
    privileged: true
    networks: { lan: { interface_name: eth0 } }
networks:
  transit: { enable_ipv4: false, enable_ipv6: true, ipam: { config: [{ subnet: "fd00:20::/64" }] } }
  ix: { enable_ipv4: false, enable_ipv6: true, ipam: { config: [{ subnet: "fd00:21::/64" }] } }
  lan: { enable_ipv4: false, enable_ipv6: true, ipam: { config: [{ subnet: "fd00:22::/64" }] } }
  e1-core: { enable_ipv4: false, enable_ipv6: true, ipam: { config: [{ subnet: "fd00:23::/64" }] } }
  e2-core: { enable_ipv4: false, enable_ipv6: true, ipam: { config: [{ subnet: "fd00:24::/64" }] } }
  e1-e2: { enable_ipv4: false, enable_ipv6: true, ipam: { config: [{ subnet: "fd00:25::/64" }] } }
  • Lines 7–17: One template for all three VyOS VMs.
  • Line 21: Each VM gets its own VM_ID, which goes into its MAC addresses. Without it, all three VMs' eth1 would share one MAC, and links between them would silently fail.
  • Line 29: core1 runs with one virtual CPU. Under software emulation on an ARM Mac, multi-vCPU guests occasionally locked up in this lab.
  • Lines 22–30: edge1 and core1 share e1-core, edge2 and core1 share e2-core, and the edges share e1-e2: the triangle.
  • Lines 51–57: Layer-2-only networks: every IPv4 address comes from the VyOS configs.
start-vyos.sh
#!/bin/sh
# Bridge every container interface (eth0, eth1, …) to a tap, then boot VyOS with one virtio NIC per tap.
# VM NIC n gets MAC 52:54:00:<VM_ID>:0n:01, so VyOS's ethN matches the container's ethN. Give each VM its own VM_ID.
set -e
NETDEVS=""
n=0
for ifc in $(ls /sys/class/net | grep -E '^eth[0-9]+$' | sort -V); do
  ip link add "br$n" type bridge
  ip link set "$ifc" master "br$n"
  ip tuntap add "tap$n" mode tap
  ip link set "tap$n" master "br$n"
  ip link set "tap$n" up
  ip link set "br$n" up
  # Pin the VM's MAC to its tap and stop learning on the Docker side: Docker reflects the VM's
  # broadcasts back, which would otherwise teach the bridge the VM's MAC on the wrong port.
  bridge link set dev "$ifc" learning off
  bridge fdb replace "52:54:00:${VM_ID:-00}:0$n:01" dev "tap$n" master static
  NETDEVS="$NETDEVS -netdev tap,id=n$n,ifname=tap$n,script=no,downscript=no -device virtio-net-pci,netdev=n$n,mac=52:54:00:${VM_ID:-00}:0$n:01"
  n=$((n + 1))
done
exec qemu-system-x86_64 -m "${VYOS_MEM:-2048}" -smp "${VYOS_SMP:-2}" -cpu max -accel tcg,thread=multi \
  -kernel /vyos/vmlinuz -initrd /vyos/initrd.img \
  -append "boot=live components hostname=vyos username=live nopersistence noautologin nonetworking union=overlay console=ttyS0,115200 net.ifnames=0 biosdevname=0" \
  -cdrom /vyos/vyos.iso $NETDEVS \
  -display none -serial unix:/run/console.sock,server=on,wait=off -monitor none
  • Lines 7–13: One bridge and tap per container interface.
  • Lines 14–17: Pin the VM's MAC to its tap and stop MAC learning on the Docker side.
  • Line 18: MAC 52:54:00:<VM_ID>:0n:01: unique per VM and per NIC.
  • Line 21: Memory and vCPU count come from the compose file.
upstream.conf (FRR)
frr defaults traditional
hostname upstream
!
interface eth0
 ip address 203.0.113.0/31
!
interface lo
 ip address 203.0.113.130/32
!
ip route 203.0.113.128/25 Null0
!
ip prefix-list CUSTOMER-PREFIXES seq 10 permit 198.51.100.0/24
!
route-map CUSTOMER-IN permit 10
 match ip address prefix-list CUSTOMER-PREFIXES
exit
!
route-map CUSTOMER-OUT permit 10
exit
!
router bgp 64496
 bgp router-id 203.0.113.0
 neighbor 203.0.113.1 remote-as 64500
 neighbor 203.0.113.1 description CUSTOMER-R1
 !
 address-family ipv4 unicast
  network 203.0.113.128/25
  neighbor 203.0.113.1 default-originate
  neighbor 203.0.113.1 route-map CUSTOMER-IN in
  neighbor 203.0.113.1 route-map CUSTOMER-OUT out
 exit-address-family
exit
!
  • Line 23: Its neighbor is now edge1, at the same address R1 used.
rs.conf (FRR)
frr defaults traditional
hostname rs1
!
interface eth0
 ip address 192.0.2.1/24
!
router bgp 64511
 bgp router-id 192.0.2.1
 no bgp ebgp-requires-policy
 neighbor 192.0.2.25 remote-as 64500
 neighbor 192.0.2.25 description R1-AS64500
 neighbor 192.0.2.40 remote-as 64502
 neighbor 192.0.2.40 description MEMBER-AS64502
 !
 address-family ipv4 unicast
  neighbor 192.0.2.25 route-server-client
  neighbor 192.0.2.40 route-server-client
 exit-address-family
exit
!
  • Line 10: edge2 takes R1's old IX address, so the route server config is unchanged.
member.conf (FRR)
frr defaults traditional
hostname member
!
interface eth0
 ip address 192.0.2.40/24
!
interface lo
 ip address 203.0.113.65/32
!
ip route 203.0.113.64/26 Null0
ip route 10.99.0.0/16 Null0
!
router bgp 64502
 bgp router-id 192.0.2.40
 no bgp ebgp-requires-policy
 no bgp enforce-first-as
 neighbor 192.0.2.1 remote-as 64511
 neighbor 192.0.2.1 description IX-RS1
 !
 address-family ipv4 unicast
  network 203.0.113.64/26
  network 10.99.0.0/16
 exit-address-family
exit
!
  • Lines 11–22: Still announcing the bogon; edge2's IX-IN drops it.

Where next

  • Route reflectors on VyOSBuilds on this lesson · Intermediate · 30 min

    Replace an iBGP full mesh with route reflectors: see iBGP refuse to pass routes on, turn on reflection, read ORIGINATOR_ID and CLUSTER_LIST, confirm the reflector stays out of the forwarding path, and survive taking one reflector down.

  • OSPF on VyOS: costs and OSPFv3Builds on this lesson · Intermediate · 30 min

    OSPF the VyOS way: interface-based configuration, reading the interface and database output, steering traffic with costs (and the asymmetric routing that follows), and OSPFv3 for IPv6 over link-local addresses. Captured on the three-router edge/core lab.

  • Setting up VyOS with basic BGPPrerequisite · Intermediate · 50 min

    Build a VyOS edge router from scratch: one upstream transit provider, one Internet Exchange route server, and the import and export policy that keeps both safe. Every command and output comes from a real VyOS lab.

  • BGP on loopbacks vs interface addressesAlso covers BGP, iBGP, OSPF · Intermediate · 45 min

    Why BGP sessions between loopbacks survive link failures and use parallel links, and what they need to work: an IGP or static routes, update-source, the eBGP connected check, and BFD. Every failure is reproduced on two real VyOS routers.