ESP Is IP Protocol 50, Not TCP Port 50: What to Permit
Protocol numbers and port numbers are different namespaces. A rule for TCP port 50 permits nothing, and NAT traversal moves ESP into UDP 4500.
Firewall forms ask for a TCP port, and the obvious entry is 50. That entry permits nothing, because ESP is IP protocol number 50 and protocol numbers are not ports. The two namespaces share small integers and nothing else, and the mistake is invisible until IKE finishes and the protected traffic never arrives.
Two namespaces that share small numbers
An IP header carries a protocol field that names what comes next: 6 for TCP, 17 for UDP, 1 for ICMP, 50 for ESP, 51 for AH. A TCP or UDP header carries port numbers, which select a process on the destination host. Protocol 50 and TCP port 50 are unrelated values that a dialog can make look interchangeable.
That is why a port-forwarding rule cannot express native ESP. There is no ESP destination port to forward, and a device that only understands port rules has no vocabulary for the packet at all. The correct rule names the IP protocol instead. Any port dialog is the wrong instrument for that packet.
The confusion has a second half. When the packet is encrypted, everything inside it is opaque, including the TCP header of the original connection. A firewall inspecting ports on that packet is looking at the outer header, which in the native case has no ports to inspect.
Native ESP, then the encapsulated alternative
Native ESP is carried directly as an IP protocol, which is efficient and awkward in equal measure. It is efficient because there is no extra header, and awkward because any device that performs address and port translation cannot track it. NAT traversal exists to solve that, by wrapping the ESP packet inside UDP so the translator has ports to rewrite.
| Form on the wire | How it is identified | What the firewall must permit |
|---|---|---|
| IKE negotiation | UDP with a destination port | UDP 500, and 4500 once traversal is detected |
| Native ESP | IP protocol 50 in the outer header | The protocol itself, with no port to name |
| ESP inside UDP | UDP 4500 carrying an ESP payload | The UDP port pair, which translation devices can track |
The two forms are alternative deployments of the same data channel, not layers of one. Allowing native ESP does not permit the encapsulated form, and allowing UDP 4500 does not permit native traffic, so a rule set has to match the deployment rather than cover both by accident.
Detection matters here as well. The endpoints decide between the forms during negotiation, and the switch usually happens when a NAT is detected between them. A path that changes after the tunnel is established can therefore move the traffic into a namespace your rules never covered.
Read which form the connection actually uses
Ask the endpoint rather than guessing from the firewall. On Linux the state entry reports whether the association is using UDP encapsulation and which ports are involved. A capture answers the same question from the wire, and both are quicker than reading a rule set and hoping.
ip xfrm state | grep -A3 encap
tcpdump -ni any 'ip proto 50 or udp port 4500'The first command shows an encapsulation type when NAT traversal is in use and nothing when it is not. The second shows which of the two forms is actually crossing the interface, which settles arguments about a configuration that looks correct on paper.
Counter evidence belongs in the same step. A rule that reports zero matches is telling you the packet never reached it, and that is a routing or selector question rather than a firewall one.
Then confirm the control channel separately. IKE has its own ports and its own logs, and a completed negotiation with no data is exactly the split this article is about.
Write the rule in the right namespace
A protocol rule and a port rule are different instructions, so the tool has to express the one you mean. In nftables the protocol match is explicit, and the encapsulated alternative gets its own UDP rule. Keep the two rules adjacent so the pair is visible in review.
nft add rule inet filter input ip protocol esp accept
nft add rule inet filter input udp dport { 500, 4500 } acceptResist widening either rule to compensate for a misdiagnosis. An allow-all for the ESP protocol on every interface, or an any-to-any UDP allowance, fixes a symptom and removes the control you were trying to configure. The narrow supported flow is almost always enough once the form is known.
Finally, keep the layers straight when the tunnel works and the application does not. Packets arriving at the gateway still need forwarding, a return route and a policy that admits them, and none of those are expressed by an ESP rule. The missing port explains a specific failure, not every IPsec failure. Keep the rule fix and the routing check as two separate conclusions.
Record the form you observed, the protocol rule you added and the counter you saw afterwards. Three lines, and the next person can tell a namespace mistake from a routing one in a minute instead of an afternoon.