strongSwan ESP-in-UDP: When There Is No NAT to Detect

NAT detection answers one question: have the addresses or ports changed in transit? It does not answer whether a firewall permits IP protocol 50. Site gateways with no translation at all can still sit behind a policy that drops native ESP, and that requirement has to be configured rather than discovered. The mechanism is ESP-in-UDP, and it is a decision rather than a fallback that appears by itself.

Detection is arithmetic on addresses, not a firewall probe

IKEv2 carries NAT-detection payloads containing hashes of the source and destination addresses and ports. Each side compares what it received against what it sees locally, and a mismatch tells it that something translated the packet. Nothing in that exchange tests whether a router in the middle forwards protocol 50.

The practical consequence is worth stating plainly. A tunnel that worked with native ESP will not discover a newly restrictive firewall by itself, and its next attempt fails at the data stage while IKE keeps succeeding. You will see established IKE security associations and no protected traffic, which looks like a routing fault until you capture on the wire.

Encapsulation is therefore a property of the connection you configure, not an outcome the peers negotiate from the network's behaviour.

How encap changes the exchange

In a modern strongSwan deployment managed through swanctl.conf, setting encap = yes on the connection influences the NAT-detection exchange so that both peers use UDP encapsulation for ESP. The protected packets then travel inside UDP, conventionally on port 4500, and both endpoints have to agree on that arrangement.

connections {
    site {
        encap = yes
        # existing approved peer, authentication and child settings
    }
}

That fragment belongs inside a real connection definition and is not a complete site profile. Keep the peer addresses, authentication and child traffic selectors you already run; the single added key changes how the protected data is wrapped, not what it protects.

The strongSwan NAT-traversal guide describes this behaviour, including the interaction with port floating when MOBIKE is in use.

Use the setting that belongs to your backend

strongSwan has two configuration backends, and the key differs between them. A swanctl.conf connection takes encap, while a legacy ipsec.conf connection takes forceencaps. Dropping one backend's keyword into the other produces a configuration that loads without complaint and changes nothing, which is worse than an error.

Check which daemon is actually running before you edit anything. On many distributions both files exist because one is a leftover from an earlier migration, and the active one is whichever the service unit starts. Read that unit rather than guessing from file timestamps. A five-second check here prevents hours of testing a setting that nothing loaded.

Apply the change through the connection owner and re-establish the association during a controlled window. An already installed child keeps its current encapsulation until it is replaced, so a file edit alone does not prove the new arrangement is in use.

Verify the wrapped data, not just the port

Capture on the gateway while protected traffic flows, then compare that with strongSwan's own view:

$ sudo tcpdump -ni eth0 'udp port 4500' -c 20
$ sudo swanctl --list-sas --ike site
$ sudo ip xfrm state list

Native ESP appears as IP protocol 50 and looks nothing like UDP. Wrapped ESP appears as UDP datagrams on 4500 whose payload is not IKE. IKE traffic on that same port is not proof of encapsulation, because MOBIKE can move IKE there while the data path stays native, so read the payload rather than the port number.

Keep path MTU in mind as well. The wrapper adds overhead, and a path that carried native ESP may need adjustments once the packets grow. Required ICMP handling should already be part of the design; encapsulation does not remove that dependency.

Accept the change when fresh application traffic inside the tunnel is visible in the capture as wrapped ESP at both ends, and when the inner service and its return route still work. Anything less is a configuration that looks right in a file.