TCP MSS Clamping on a Linux IPsec Gateway: What It Fixes

Ping works, SSH works, a small page loads, and then every large response hangs until it times out. The classic pattern points at packet size rather than reachability: the path carries the handshake and the first few kilobytes, then a full-size segment disappears. On a Linux IPsec gateway, a TCP MSS rule is often the fix, but only if you understand what it changes and where the packet it touches actually is. The order below follows the packet from symptom to rule placement, so you can stop as soon as the explanation stops fitting.

The symptom that suggests MSS

TCP negotiates a maximum segment size in the handshake. If that value is larger than the path can carry after encapsulation, the sender keeps producing segments that never arrive, and the connection stalls after the first round trip. Small responses fit; large ones do not.

ICMP feedback can rescue the connection through path MTU discovery, and a firewall that drops those messages turns the stall into a silent black hole. That is why the same tunnel works for one application and fails for another with no obvious difference in configuration.

What a TCPMSS rule does and does not do

The rule rewrites the MSS option in TCP handshake packets as they pass. It does not set the tunnel interface's MTU, it does not change the IPsec policy, and on current kernels it only lowers an advertised value rather than raising one that is already smaller. It also only touches TCP; UDP payloads, DNS behaviour and the negotiated traffic selectors are outside its reach.

Because the change happens during the handshake, an existing connection that was established before the rule keeps its old value. Test with a new connection, and expect the first attempt after the change to be the one that tells you anything. Nothing in the rule reaches a connection that is already open. That is why a fix applied in the middle of an incident can look like it did nothing at all.

Put the rule on the inner TCP path

On a policy-based Linux gateway, the forwarded packet still looks like ordinary TCP when the firewall sees it. The IPsec policy decides that the packet will be protected; the firewall does not see an ESP header for the inner flow, so matching on ESP here describes a different packet entirely. That mismatch is why some rules appear correct and never match.

In iptables terms, the rule belongs in the mangle table on the forwarding path, restricted to handshake packets: iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1270. The number is an example, not a universal constant; derive it from the encapsulation overhead and the path's real MTU. An nftables equivalent sets the max segment size in a forward or postrouting chain.

Keep the outer listener rules for IKE and ESP separate from the inner forwarding rules. A single rule that tries to authorise both the application flow and the outer protocol ends up matching neither reliably.

Respect the firewall's owner and the NAT order

If UFW owns the host firewall, add the mangle and forwarding changes through its supported persistent configuration rather than by typing a command that disappears at the next reload. A rule inserted in the wrong table or after a terminal rule has no effect, and the live table hides that from you.

Source translation is a separate decision that often interacts with this one. An exemption placed before a general masquerade rule keeps protected traffic from losing the source its policy requires; it does not repair a size problem, and a size rule does not repair a translation problem.

Verify narrowly, then stop

Open a fresh connection to the failing service and watch both the application and the rule counters. If the stall clears, you have evidence about that TCP path with that encapsulation. You do not have evidence that every VPN protocol fits, that every UDP application is fixed, or that a specific upstream service was the original cause.

Leave ICMP size feedback working where policy allows, and treat automatic clamping based on route MTU with care when the outer IPsec path is not represented by the inner route. The narrower the conclusion, the easier the next incident is to diagnose.