Fair between hosts
Shaping fixes latency. CAKE also decides who gets the bandwidth, and that's where the lab held a surprise.
Two office PCs upload at once: client with one connection, client2 with eight (as a backup tool or a download manager might). With a plain queue, each connection gets an equal share, so client2 takes eight ninths: in the lab, 4.39 Mbit/s against 471 kbit/s.
CAKE's default isolation mode, triple-isolate, is meant to share fairly between hosts. In this lab it didn't: client still got 471 kbit/s, and client2 3.46 Mbit/s. Both PCs talk to the same server, and triple-isolate's mix of source-host and destination-host fairness didn't separate them. Telling CAKE exactly what to share by fixed it:
- Uploads (egress on the WAN):
dual-srchost, fairness between the office's source addresses. Both PCs then got exactly 2.10 Mbit/s. - Downloads (the IFB):
dual-dsthost, fairness between destination addresses. One connection against eight went from 1.83 vs 13.6 Mbit/s to 8.55 vs 6.45 Mbit/s.
Within each host, CAKE still shares fairly between that host's connections.
With NAT
On a typical office router, the WAN interface masquerades the LAN. CAKE on the WAN then only sees one source address, the router's, and can't tell the PCs apart. The nat keyword (cake bandwidth 4500kbit dual-srchost nat) makes CAKE look up each packet's original address in the connection tracking table first. Use it whenever the shaped interface does NAT. (This lab routes the office's addresses instead, so it doesn't need it.)
root@gw:~# tc qdisc replace dev ens4 root fq_codel && tc qdisc show dev ens4 | cut -c1-30
qdisc fq_codel 8003: root refc
qdisc ingress ffff: parent fff
root@client2:~# nohup iperf3 -c 198.51.100.10 -p 5202 -P 8 -t 20 > /tmp/eight.txt 2>&1 &
root@client:~# iperf3 -c 198.51.100.10 -t 20 | grep receiver
[ 5] 0.00-20.10 sec 1.12 MBytes 469 Kbits/sec receiver
root@client2:~# grep "SUM.*receiver" /tmp/eight.txt
[SUM] 0.00-21.03 sec 11.0 MBytes 4.39 Mbits/sec receiver
root@gw:~# tc qdisc replace dev ens4 root cake bandwidth 4500kbit && tc qdisc show dev ens4 | cut -c1-60
qdisc cake 8004: root refcnt 2 bandwidth 4500Kbit diffserv3
qdisc ingress ffff: parent ffff:fff1 ----------------
root@client2:~# nohup iperf3 -c 198.51.100.10 -p 5202 -P 8 -t 20 > /tmp/eight.txt 2>&1 &
root@client:~# iperf3 -c 198.51.100.10 -t 20 | grep receiver
[ 5] 0.00-20.02 sec 1.12 MBytes 471 Kbits/sec receiver
root@client2:~# grep "SUM.*receiver" /tmp/eight.txt
[SUM] 0.00-20.01 sec 8.25 MBytes 3.46 Mbits/sec receiver
root@gw:~# tc qdisc replace dev ens4 root cake bandwidth 4500kbit dual-srchost && tc qdisc show dev ens4 | cut -c1-75
qdisc cake 8004: root refcnt 2 bandwidth 4500Kbit diffserv3 dual-srchost no
qdisc ingress ffff: parent ffff:fff1 ----------------
root@client2:~# nohup iperf3 -c 198.51.100.10 -p 5202 -P 8 -t 20 > /tmp/eight.txt 2>&1 &
root@client:~# iperf3 -c 198.51.100.10 -t 20 | grep receiver
[ 5] 0.00-20.01 sec 5.00 MBytes 2.10 Mbits/sec receiver
root@client2:~# grep "SUM.*receiver" /tmp/eight.txt
[SUM] 0.00-20.01 sec 5.00 MBytes 2.10 Mbits/sec receiver- Lines 1–11: A plain queue: client gets 471 kbit/s, client2's eight connections 4.39 Mbit/s.
- Lines 13–23: CAKE with its default triple-isolate: no better here.
- Lines 25–35: CAKE with dual-srchost: exactly 2.10 Mbit/s each.
root@gw:~# tc qdisc show dev ifb4ens4 | cut -c1-75
qdisc cake 8002: root refcnt 2 bandwidth 18Mbit diffserv3 triple-isolate no
root@client2:~# nohup iperf3 -c 198.51.100.10 -p 5202 -P 8 -t 20 -R > /tmp/eight.txt 2>&1 &
root@client:~# iperf3 -c 198.51.100.10 -t 20 -R | grep receiver
[ 5] 0.00-20.00 sec 4.38 MBytes 1.83 Mbits/sec receiver
root@client2:~# grep "SUM.*receiver" /tmp/eight.txt
[SUM] 0.00-20.00 sec 32.4 MBytes 13.6 Mbits/sec receiver
root@gw:~# tc qdisc replace dev ifb4ens4 root cake bandwidth 18mbit ingress dual-dsthost && tc qdisc show dev ifb4ens4 | cut -c1-75
qdisc cake 8002: root refcnt 2 bandwidth 18Mbit diffserv3 dual-dsthost nona
root@client2:~# nohup iperf3 -c 198.51.100.10 -p 5202 -P 8 -t 20 -R > /tmp/eight.txt 2>&1 &
root@client:~# iperf3 -c 198.51.100.10 -t 20 -R | grep receiver
[ 5] 0.00-20.00 sec 20.4 MBytes 8.55 Mbits/sec receiver
root@client2:~# grep "SUM.*receiver" /tmp/eight.txt
[SUM] 0.00-20.00 sec 15.4 MBytes 6.45 Mbits/sec receiver- Lines 1–10: Default triple-isolate on the IFB: 1.83 against 13.6 Mbit/s.
- Lines 12–21: dual-dsthost: 8.55 against 6.45 Mbit/s.
Check your understanding
The router masquerades the LAN behind its WAN address, and you shape the WAN with dual-srchost. Why does one PC still take most of the upload?
- After NAT, every packet has the router's address as its source; add
natso CAKE looks up the original address in conntrack. - dual-srchost only works on downloads.
- CAKE can't run on an interface that does NAT.
- The PC has a faster network card.