Traceroute across the internet
A trace across your own lab is tidy. A trace across the internet almost never is: you'll see hops with * * *, hops that answer with two or three different addresses, and round-trip times that go down as the path gets longer. Almost none of that means something is broken. Knowing which patterns are harmless is most of the skill.
Below is a real traceroute from a home connection in Australia to a web server in North America. It was captured with traceroute -n on macOS: Docker's networking can't run traceroute out to the internet, and with -n the output format is the same as Linux's, apart from the probe size. Every address has been replaced with a documentation address, consistently, so the same router always has the same replacement. The timings and every * are exactly as captured.
Step through it, then use these rules:
- A silent hop in the middle of a working path doesn't matter. If later hops answer, packets are passing through it.
- One slow hop doesn't matter either, unless the extra delay persists to every hop after it. Routers answer traceroute with their CPU at low priority, not with their forwarding hardware.
- Several addresses on one hop is load balancing, not a routing problem.
* * *at the end of a trace usually means "filtered", not "down". Test the actual service before you conclude anything.
Interactive: Reading a real internet traceroute A real trace from Australia to North America, 16 answering hops and then silence. Click each hop to see what it tells you. (needs JavaScript)
user@mac ~ % traceroute -I -n -q 3 -w 2 -m 25 192.0.2.47
traceroute to 192.0.2.47 (192.0.2.47), 25 hops max, 48 byte packets
[... hops 1-13 trimmed: same path as the UDP trace ...]
14 203.0.113.33 236.504 ms 236.325 ms 284.489 ms
15 203.0.113.30 238.490 ms 238.304 ms 238.259 ms
16 192.0.2.141 238.132 ms 238.193 ms 238.231 ms
17 192.0.2.198 228.752 ms 234.714 ms 229.663 ms
18 * * *
19 * * *
[... hops 20-25 trimmed: * * * ...]
user@mac ~ % nc -vz -w 5 192.0.2.47 443
Connection to 192.0.2.47 port 443 [tcp/https] succeeded!
user@mac ~ % ping -c 3 -W 2000 192.0.2.47
[... "PING" header and three "Request timeout" lines trimmed ...]
--- 192.0.2.47 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss- Lines 1–2: The same trace using ICMP echo probes (
-I) instead of UDP. - Line 7: ICMP probes reach one hop further than UDP probes did: the destination network's own firewall.
- Lines 8–10: Then silence again: the server doesn't answer ICMP either.
- Lines 11–12: Yet a TCP connection to the HTTPS port of exactly the same address succeeds. The server is up; it just filters probes.
- Lines 13–16: And ping fails, consistent with the trace. A host that ignores ping is not necessarily down.
Check your understanding
A traceroute to a website shows normal hops 1–12, then * * * until the 30-hop limit. The website loads fine in a browser. What's going on?
- The website is down from hop 13 onwards
- The destination network drops traceroute's probes (UDP or ICMP), but allows the web traffic
- There's a routing loop after hop 12
- Hop 12 is rate-limiting ICMP