MSS Clamping on a VPN Gateway: What It Fixes and What It Does Not
Clamping rewrites the MSS in TCP handshakes so segments fit the tunnel. It never fixes UDP or a broken path-MTU check. Three checks, in order.
Small requests work, large downloads hang, and somebody suggests MSS clamping. That suggestion is right often enough to be worth trying in the correct order, and wrong often enough to waste an afternoon if you apply it blind. The three checks below separate a genuine TCP sizing problem from a UDP problem, a broken path-MTU discovery, and a clamp that never sees the handshake it is supposed to rewrite.
Start with the shape of the failure
A sizing fault has a signature: the connection establishes, authentication and small responses succeed, and then a transfer stalls indefinitely once it needs a full-size segment. Pages that load text but never finish an image are the same failure at a smaller scale. If nothing connects at all, the problem is earlier in the path and clamping will not help. A clamp cannot repair a handshake that never arrived.
The signature exists because the handshake is where the segment size is agreed. Each side advertises a maximum segment size as a TCP option, and senders respect the peer's advertisement. If the path cannot carry a segment that large, the failure appears later, when the sender finally tries one.
Keep the transport in view while reading that symptom. Clamping is a TCP option rewrite, so a voice or video call that stalls under the same tunnel is not an MSS problem and will not be improved by one. Write the transport down before you touch a TCP option.
Price the path, then compare it with the tunnel
Two numbers matter and they are often confused. The path MTU is the largest packet the path will carry without fragmentation, and the advertised MSS for IPv4 is that value minus the 40 bytes of IP and TCP headers, or minus 60 for IPv6. Tunnel encapsulation consumes part of the budget before the inner packet even begins.
ping -M do -s 1400 198.51.100.20
tracepath 198.51.100.20The ping with the do-not-fragment bit tells you whether a packet of that payload size survives; adjust the size until it fails and you have found the path limit empirically. The tracepath output reports the MTU discovered along the route, which is faster when it works and less reliable when an intermediate hop filters the probes.
Now subtract the tunnel's overhead. An outer IP header, a UDP header and a tunnel-specific header all come out of the same 1500 bytes, so the usable inner MTU on a typical Ethernet underlay is lower than the physical link suggests. If the computed inner MTU is below what your peers advertise, you have found a real clamp candidate rather than a guess. If the computed value is already above what peers advertise, stop here and look elsewhere.
Place the clamp where the handshake passes
A clamp works by rewriting the MSS option in the SYN and SYN-ACK packets as they cross the gateway. If the rule sits on an interface that either direction never traverses, the negotiation completes unchanged and the symptom persists, which is why placement decides the outcome more than the value does.
| Symptom | What it points to | Next check |
|---|---|---|
| Handshake completes, large transfer stalls | Segment size larger than the path can carry | Path MTU against the tunnel's effective MTU |
| Nothing connects, small or large | Routing, firewall or keying failure | Not MSS; look at the tunnel state |
| Only UDP applications stall | Datagram size or a dropped ICMP error | Platform-specific UDP handling, not clamping |
| Fixed after reconnect, returns later | Clamp applied to one direction or one interface | Rule placement and the handshake path |
Direction is the other half. Clamping one direction fixes half the flows and leaves the rest failing in a way that looks intermittent, so apply it to both directions of the forwarded traffic. The classic Linux form for a tunnel interface is a mangle rule matching SYN packets:
iptables -t mangle -A FORWARD -o tun0 -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtuNote what --clamp-mss-to-pmtu actually does: it derives the value from the route's MTU, which is not always the tunnel's real payload budget. On a route-based tunnel where the inner route and the outer path differ, an explicit value computed in the previous check is more predictable than an automatic one.
Verify with a new connection, and keep the feedback alive
An established connection keeps the MSS it negotiated, so testing on the same session proves nothing. Open a fresh connection after the change and read the negotiated value directly rather than inferring it from a transfer that happens to finish.
ss -tin dst 198.51.100.20
iptables -t mangle -S | grep TCPMSSThe first command reports the socket's MSS alongside its congestion state, which is the fastest confirmation that the clamp took effect. The second confirms the rule you think you added is the rule that exists, in the table where it was placed.
Then protect the mechanism you did not fix. Path-MTU discovery depends on ICMP fragmentation-needed messages reaching the sender, so a firewall that drops them will keep breaking large transfers even with a correct clamp in place. Allow those messages for the tunnel path, or accept that clamping is compensating for a policy you chose.
Last, resist the temptation to treat the clamp as a tuning knob. It constrains segment size, which costs throughput on paths that never needed it. Applied after the three checks it is a targeted repair; applied first it is a change that hides the original fault and adds a second variable to every later test. That trade is worth making deliberately, not by default.
Write down the path MTU you measured, the tunnel overhead you subtracted, the rule you added and the MSS you saw afterwards. Four numbers make the next incident a lookup instead of an investigation.