IKE over TCP 4500: check both peers before the port rule
TCP encapsulation is negotiated by both peers, not granted by a firewall rule. Run the listener check, the capture and the four acceptance stages in order.
Support tickets often arrive with the firewall change already approved and the tunnel still refusing to come up. That is the expected result when only one end has the feature: RFC 9329 TCP encapsulation has to be implemented and configured on both peers, and the agreement happens inside the connection before any protected traffic can move. Opening a port advertises a listener that may not exist, and a listener without a matching configuration on the other side accepts a connection and then goes quiet. The sequence below puts the cheap checks first.
Know what the option actually adds
Native IPsec carries encrypted payloads as IP protocol 50, with no port number to permit. NAT traversal wraps those packets in UDP to port 4500, which is why so many firewall rules mention that port. RFC 9329 adds a stream representation: IKE and IPsec packets both travel inside one TCP connection, normally to port 4500.
The stream is not a web service. A TCP connection to 4500 says nothing about HTTP, TLS or proxy friendliness, and an HTTP proxy in the path will usually break the framing rather than carry it.
Preflight before you touch the firewall
The standard positions TCP as a fallback for networks that drop the UDP path, not as a replacement for it. Get two facts from each vendor: the exact feature name and the version that implements RFC 9329 rather than a proprietary variant. Labels such as IPsec over TCP have been used for unrelated designs, so ask for the document reference and the configuration key that turns it on. Without both answers, assume nothing about wire compatibility.
Prove it in a lab where both ends are yours. A gateway that answers on TCP 4500 but never completes IKE is the signature of a listener without the matching encapsulation on the other side, and it is much cheaper to discover there than during a change window.
On the responder: is anything listening
Start with the socket. ss -ltnp | grep 4500 shows whether a TCP listener exists and which process owns it. The absence of a line ends the investigation at the gateway configuration rather than at the network.
If the socket is there, watch the handshake instead of guessing from the client side: tcpdump -ni any 'tcp port 4500'. A completed three-way handshake followed by silence means the stream was accepted and the IKE exchange inside it failed, while a SYN with no answer points at filtering or at the wrong address.
Compare against the UDP path in the same capture window so the two are not confused: tcpdump -ni any 'udp port 4500 or proto 50'. Traffic on protocol 50 or UDP port 4500 while you believe TCP is in use means the peers never agreed on the fallback. Capture on the gateway as well if you can, because a client-side capture cannot show what the far end sent.
Accept in four separate stages
Treat the bring-up as four results. Each can fail for a different reason, and a single pass or fail hides which one did, so stop at the first stage that fails and start again from the top.
| Stage | What you check | What it proves |
|---|---|---|
| 1 | TCP connection established to 4500 | A listener exists and the path permits it |
| 2 | IKE_SA authenticated in the gateway log | Both peers agreed on the encapsulation |
| 3 | CHILD_SA installed, visible with ip xfrm state | Keys exist for the protected traffic |
| 4 | A new application exchange succeeds | The data path works, not just the control path |
Stage four is the one that gets skipped. An existing session may keep working on cached state, so open a fresh connection to a host behind the gateway before closing the change. If stage two fails while stage one passes, the disagreement sits in the proposal or in the encapsulation setting, not in the firewall.
Rollback, and the cost of leaving it on
Plan the undo before the change: turn the encapsulation option off at both peers and confirm the tunnel returns over UDP. Keep the TCP rule for one review cycle in case the fallback is needed again, then remove it. An unused listener on 4500 is surface area with no operational value.
Watch for the performance signature once it carries real work. TCP inside TCP adds acknowledgement layers that stall on a single loss, and streams sharing one connection can block each other, so measure the application that will use the tunnel instead of a ping.
Acceptance list
Confirm the feature name and version at both ends, the listener on the responder, the capture showing a completed handshake, and the four stages recorded with timestamps. Add the fallback reason to the change record so the next engineer knows why TCP is there. Rehearse the rollback before you need it.
If any stage fails, fix that stage only. Changing the cipher suite or the identity settings to chase a transport problem usually adds a second, unrelated failure on top of the first.