What is TCP-MD5?

TCP-MD5 (TCP MD5 Signature Option) adds a keyed MD5 hash to every TCP segment so that BGP routers can reject spoofed packets on their sessions.

BGP runs over a long-lived TCP connection on port 179, and anyone who can guess the addresses and ports involved could try to inject a reset or fake data into it. TCP-MD5, defined in RFC 2385, defends against that. Both routers are configured with the same shared password. For every segment it sends, the sender computes an MD5 digest over parts of the IP and TCP headers, the payload and the password, and carries the result in a TCP option. The receiver repeats the calculation and silently discards any segment whose digest doesn't match, before TCP processes it. An attacker without the password can't produce valid segments, so blind resets and injection stop working. On most platforms this is the password setting on a BGP neighbour, and on Linux it's implemented by the kernel, which FRR and VyOS use.

TCP-MD5 is still the most widely deployed BGP session protection, and many providers send a password along with your session details. It is weak by modern standards, though: MD5 is an old algorithm, and there's no way to change the key without disrupting the session, so passwords tend to stay the same for years. RFC 5925 replaced it with TCP-AO, which supports stronger algorithms and key rollover, but support is still uneven. The usual mistake is a mismatched or mistyped password: because bad segments are dropped silently, the session just never comes up, with no clear error from BGP itself, though the kernel often logs a hash mismatch. Many networks combine it with GTSM, which rejects packets that didn't come from a directly connected neighbour. The BGP Fundamentals lesson covers session protection alongside other BGP security measures.