TTR Networks

Intermediate · Packets That Never Come Back · section 6 of 7

Scenario 3: MTU black hole

✓ Lab-tested 17 October 2026 on VyOS 2026.10.16-0024-rolling, Debian 12

Symptom: after R1's WAN link moves to an encapsulated uplink (MTU 1400: PPPoE, VPN tunnels and some cloud links have smaller MTUs), ping works, connections open, small requests work, but large uploads hang.

Packets on a path can't exceed the smallest MTU along it. TCP avoids fragmentation with path MTU discovery: it sends full-size packets with the Don't Fragment (DF) bit set, and any router that can't forward one replies with ICMP "fragmentation needed" quoting the MTU it can handle. The sender lowers its packet size and retransmits.

Investigation:

  1. In the healthy case, a capture with -v shows R1 answering each oversized DF segment with ICMP need to frag (mtu 1400). Afterwards, ip route get on the client shows a learned mtu 1400 in the route cache, and the upload completes.
  2. In the broken case, the client's own firewall drops incoming ICMP "destination unreachable", a common over-zealous hardening rule. The capture shows the ICMP arriving, because tcpdump sees it before the firewall drops it. Yet the client keeps retransmitting the same 1448-byte segment with growing back-off, the route cache never learns the MTU, and the server receives 0 bytes.

That pattern, small packets fine, large ones vanish, ICMP unreachables ignored, is an MTU black hole.

Fixes:

  • Best: don't filter ICMP type 3 ("destination unreachable", code 4 = fragmentation needed). Path MTU discovery depends on it.
  • Router-side: MSS clamping. R1 rewrites the Maximum Segment Size option in SYN/SYN-ACK packets so both ends agree on segments that fit. In VyOS that's ip adjust-mss on an interface. It applies to SYNs leaving that interface, so to protect clients' uploads, clamp on the LAN side (eth0), where the server's SYN-ACK leaves toward the client.
Diagram: Scenario 3: MTU black hole — Small packets fine, large ones vanish, and how MSS clamping avoids the whole problem.
  1. Small packets work. The handshake and small requests fit any MTU, so ping and connection tests all succeed.
  2. A full-size segment can't fit. The client sends 1500-byte packets with Don't Fragment set. R1's WAN MTU is 1400, so it can't forward them.
  3. R1 says so with ICMP. R1 sends ICMP "fragmentation needed, MTU 1400" back to the client. Normally the client lowers its path MTU and resends: path MTU discovery.
  4. …but the ICMP is dropped. The client's firewall drops ICMP unreachables. It never learns, retransmits the same 1500-byte packet with growing back-off, and the transfer hangs.
  5. MSS clamping on R1. R1 rewrites the MSS in SYN-ACKs leaving eth0 to 1360, so the client never sends anything too big. The transfer works even with ICMP still blocked.
Healthy: path MTU discovery working (real output, trimmed)
root@client:~# timeout 5 tcpdump -l -nn -v -i eth0 icmp
tcpdump: listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
21:41:34.199639 IP (tos 0xc0, ttl 64, id 3437, offset 0, flags [none], proto ICMP (1), length 576)
    10.0.0.1 > 10.0.0.10: ICMP 203.0.113.20 unreachable - need to frag (mtu 1400), length 556
	IP (tos 0x0, ttl 64, id 33248, offset 0, flags [DF], proto TCP (6), length 1500)
    10.0.0.10.43200 > 203.0.113.20.9000: Flags [.], seq 1780747227:1780748675, ack 1913778425, win 63, options [nop,nop,TS val 982827524 ecr 180258314], length 1448
21:41:34.199675 IP (tos 0xc0, ttl 64, id 3438, offset 0, flags [none], proto ICMP (1), length 576)
    10.0.0.1 > 10.0.0.10: ICMP 203.0.113.20 unreachable - need to frag (mtu 1400), length 556
	IP (tos 0x0, ttl 64, id 33249, offset 0, flags [DF], proto TCP (6), length 1500)
    10.0.0.10.43200 > 203.0.113.20.9000: Flags [.], seq 1448:2896, ack 1, win 63, options [nop,nop,TS val 982827524 ecr 180258314], length 1448
[... 32 lines trimmed: the same ICMP message for each of the next 8 segments ...]

10 packets captured
10 packets received by filter
0 packets dropped by kernel
  • Lines 3–4: R1 (10.0.0.1) tells the client the destination is unreachable unless it uses packets of at most 1400 bytes.
  • Lines 5–6: The ICMP message quotes the packet that didn't fit: 1500 bytes, flags [DF], 1448 bytes of TCP payload.
The client learned the MTU (real output)
root@client:~# ip route get 203.0.113.20
203.0.113.20 via 10.0.0.1 dev eth0 src 10.0.0.10 uid 0
    cache expires 599sec mtu 1400
  • Line 3: A route-cache entry: use MTU 1400 toward this destination for the next ~10 minutes. The upload finished.
Now the client's firewall drops ICMP unreachables
root@client:~# nft list table inet filter
table inet filter {
	chain input {
		type filter hook input priority filter; policy accept;
		icmp type destination-unreachable drop
	}
}
  • Line 5: A typical "harden ICMP" rule. It also throws away "fragmentation needed", which is ICMP type 3, code 4.
The black hole (real output, trimmed)
root@client:~# timeout 8 tcpdump -l -nn -i eth0 "icmp or (tcp port 9000 and greater 1400)"
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
21:41:54.845956 IP 10.0.0.10.48270 > 203.0.113.20.9000: Flags [.], seq 17483820:17485268, ack 764479682, win 63, options [nop,nop,TS val 741302563 ecr 2877791054], length 1448
21:41:54.845957 IP 10.0.0.10.48270 > 203.0.113.20.9000: Flags [.], seq 1448:2896, ack 1, win 63, options [nop,nop,TS val 741302563 ecr 2877791054], length 1448
21:41:54.845957 IP 10.0.0.10.48270 > 203.0.113.20.9000: Flags [.], seq 2896:4344, ack 1, win 63, options [nop,nop,TS val 741302563 ecr 2877791054], length 1448
21:41:54.845957 IP 10.0.0.10.48270 > 203.0.113.20.9000: Flags [.], seq 4344:5792, ack 1, win 63, options [nop,nop,TS val 741302563 ecr 2877791054], length 1448
21:41:54.845958 IP 10.0.0.10.48270 > 203.0.113.20.9000: Flags [P.], seq 5792:7240, ack 1, win 63, options [nop,nop,TS val 741302563 ecr 2877791054], length 1448
21:41:54.845972 IP 10.0.0.10.48270 > 203.0.113.20.9000: Flags [.], seq 7240:8688, ack 1, win 63, options [nop,nop,TS val 741302563 ecr 2877791054], length 1448
21:41:54.845973 IP 10.0.0.10.48270 > 203.0.113.20.9000: Flags [.], seq 8688:10136, ack 1, win 63, options [nop,nop,TS val 741302563 ecr 2877791054], length 1448
21:41:54.845973 IP 10.0.0.10.48270 > 203.0.113.20.9000: Flags [.], seq 10136:11584, ack 1, win 63, options [nop,nop,TS val 741302563 ecr 2877791054], length 1448
21:41:54.845973 IP 10.0.0.10.48270 > 203.0.113.20.9000: Flags [.], seq 11584:13032, ack 1, win 63, options [nop,nop,TS val 741302563 ecr 2877791054], length 1448
21:41:54.845973 IP 10.0.0.10.48270 > 203.0.113.20.9000: Flags [P.], seq 13032:14480, ack 1, win 63, options [nop,nop,TS val 741302563 ecr 2877791054], length 1448
21:41:54.846332 IP 10.0.0.1 > 10.0.0.10: ICMP 203.0.113.20 unreachable - need to frag (mtu 1400), length 556
[... 17 lines trimmed: more retransmissions, each answered by an ICMP the client drops ...]
21:41:57.974574 IP 10.0.0.10.48270 > 203.0.113.20.9000: Flags [.], seq 0:1448, ack 1, win 63, options [nop,nop,TS val 741305692 ecr 2877791054], length 1448
21:41:57.975417 IP 10.0.0.1 > 10.0.0.10: ICMP 203.0.113.20 unreachable - need to frag (mtu 1400), length 556
21:42:01.303143 IP 10.0.0.10.48270 > 203.0.113.20.9000: Flags [.], seq 0:1448, ack 1, win 63, options [nop,nop,TS val 741309020 ecr 2877791054], length 1448
21:42:01.305799 IP 10.0.0.1 > 10.0.0.10: ICMP 203.0.113.20 unreachable - need to frag (mtu 1400), length 556

32 packets captured
32 packets received by filter
0 packets dropped by kernel
  • Lines 4–13: The client sends a window of full-size (1448-byte payload) segments.
  • Line 14: R1's ICMP does arrive: tcpdump sees it before nftables drops it on input.
  • Lines 16–19: So the client keeps retransmitting the first segment with exponential back-off (~3 s, then more), and R1 keeps answering. It never ends.
No MTU learned (real output)
root@client:~# ip route get 203.0.113.20
203.0.113.20 via 10.0.0.1 dev eth0 src 10.0.0.10 uid 0
    cache
  • Line 3: Compare with the healthy case: no mtu 1400. The information was dropped.
Router fix: MSS clamping, seen at the client (real output)
root@client:~# timeout 6 tcpdump -l -nn -c 3 -i eth0 "tcp port 9000 and tcp[tcpflags] & tcp-syn != 0"
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
21:42:52.093664 IP 10.0.0.10.36054 > 203.0.113.20.9000: Flags [S], seq 2571787087, win 64240, options [mss 1460,sackOK,TS val 2097017950 ecr 0,nop,wscale 10], length 0
21:42:52.097511 IP 203.0.113.20.9000 > 10.0.0.10.36054: Flags [S.], seq 452381738, ack 2571787088, win 65160, options [mss 1360,sackOK,TS val 2409830211 ecr 2097017950,nop,wscale 10], length 0

2 packets captured
2 packets received by filter
0 packets dropped by kernel
  • Line 1: Configured on R1 with set interfaces ethernet eth0 ip adjust-mss 1360. Then: show only SYN and SYN-ACK packets.
  • Line 4: The client offers MSS 1460, based on its own 1500-byte MTU.
  • Line 5: The SYN-ACK arrives offering MSS 1360, so the client will never send more than 1360 + 40 = 1400 bytes.
…and at the server: it really sent 1460 (real output)
root@server:~# timeout 6 tcpdump -l -nn -c 2 -i eth0 "tcp port 9000 and tcp[tcpflags] & tcp-syn != 0"
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
21:43:11.224417 IP 203.0.113.1.56860 > 203.0.113.20.9000: Flags [S], seq 185527964, win 64240, options [mss 1460,sackOK,TS val 3899156571 ecr 0,nop,wscale 10], length 0
21:43:11.224431 IP 203.0.113.20.9000 > 203.0.113.1.56860: Flags [S.], seq 2049358249, ack 185527965, win 65160, options [mss 1460,sackOK,TS val 1099342265 ecr 3899156571,nop,wscale 10], length 0
2 packets captured
2 packets received by filter
0 packets dropped by kernel
  • Line 4: With NAT, the server sees R1's address as the source.
  • Line 5: The server's SYN-ACK says 1460. R1 rewrote it to 1360 on its way out of eth0.
The nftables rule VyOS generated (real output)
vyos@R1:~$ sudo nft list chain ip raw VYOS_TCP_MSS
table ip raw {
	chain VYOS_TCP_MSS {
		type filter hook postrouting priority raw; policy accept;
		oifname "eth0" tcp flags syn / syn,rst tcp option maxseg size 1361-65535 tcp option maxseg size set 1360
	}
}
  • Line 5: For SYN packets leaving eth0 with an MSS above 1360, set it to 1360. That's why the clamp goes on the interface the SYN-ACK leaves by.
Upload results, in order (real output)
root@server:~# cat /tmp/sink.log
received 200000 bytes from 203.0.113.1
received 200000 bytes from 203.0.113.1
received 0 bytes from 203.0.113.1
received 200000 bytes from 203.0.113.1
received 200000 bytes from 203.0.113.1
  • Lines 2–3: Path MTU discovery working: complete.
  • Line 4: Black hole: nothing usable arrived before the server gave up.
  • Lines 5–6: With MSS clamping on R1, complete again, even though the client still drops ICMP.

Check your understanding

Users behind a new VPN can browse small pages, but large downloads stall. ping works. What's your first test?

  • Check DNS
  • ping with DF set and a full-size packet (ping -M do -s 1472), and capture for ICMP need-to-frag
  • Increase the TCP window size
  • Restart the VPN