How to read the first IKEv2 exchange in a packet capture

Mid-session captures are the usual reason an IKEv2 trace looks like noise, because the peers are already rekeying or exchanging encrypted payloads when recording begins. Start from a fresh connection attempt instead, so the first message in the file is the one that opens the negotiation. The reading order below assumes that, and it marks the point where a packet stops proving anything about the far end. Skipping that step is how a working tunnel gets diagnosed as a blocked port.

Confirm you are looking at the opening exchange

An IKEv2 negotiation begins with IKE_SA_INIT, exchange type 34, message ID 0. The initiator picks a nonzero SPI, and the responder SPI is zero in that first request and nonzero in the reply. Those three facts separate a fresh attempt from a rekey or from an IKE_AUTH message that happens to sit at the top of a capture.

Header fields are readable, and so are the proposal, key exchange and nonce payloads. Readable is not authenticated: neither peer has proven identity at this point, and the Diffie-Hellman private value never appears on the wire.

Print the fields as a table rather than scrolling a decode pane, because retransmissions and mismatched SPIs become obvious in columns.

tshark -r ike.pcapng -Y "isakmp" -T fields -e frame.number -e ip.src -e udp.srcport -e isakmp.exchangetype -e isakmp.messageid -e isakmp.initiator_spi -e isakmp.responder_spi

Read the output in pairs. A request with message ID 0 followed by a reply carrying the same ID and a nonzero responder SPI is one exchange, even when it appears three times because the request was retransmitted. Three identical requests are not three logins, and treating them as such is how people conclude a client is looping when it is merely patient. Sort by message ID, then by timestamp, and the whole conversation collapses into a short list of attempts.

Read a rejection before you read a success

A reply is not automatically progress. The responder may answer with a cookie notification and expect the initiator to retry, or with INVALID_KE_PAYLOAD naming a group it would accept instead. Both arrive as well-formed IKE messages, and both look like a finished exchange to a filter that only counts packets.

Open the notify payload and read the type before anything else. A cookie challenge means the responder is defending itself against spoofed source addresses, and the client must resend with the cookie attached. An invalid group means the two proposals disagree, which no amount of retrying will change until one side's configuration moves.

Only IKE_AUTH, the second exchange, carries the encrypted identity and the authentication payloads. A completed IKE_SA_INIT therefore tells you the peers agree on algorithms and derived keys, and nothing at all about whether the credentials will be accepted. Expect two messages there as well, and expect both to be opaque without the session keys.

Port 4500 is not one protocol

UDP 500 usually carries the opening exchange. Once NAT detection has run, IKE moves to 4500 and prefixes each message with a four-byte zero non-ESP marker. Encapsulated ESP on the same port begins with a nonzero ESP SPI instead, and a single-byte NAT keepalive is a third shape that appears when nothing else is happening.

A filter that matches the port alone mixes all three, which is why "there is traffic on 4500" settles so little. Split the view by protocol before you conclude anything about the peer.

tshark -r ike.pcapng -Y "udp.port == 4500 && isakmp" -T fields -e frame.number -e isakmp.exchangetype
tshark -r ike.pcapng -Y "udp.port == 4500 && !isakmp" -T fields -e frame.number -e esp.spi

Silence describes the observation point

Requests without replies establish exactly one thing: at that capture point, no reply appeared. Filtering, a route that sends the answer elsewhere, an unreachable responder, or a responder that dislikes the proposal all produce the same empty result, and a generic UDP probe will not separate them.

An ICMP unreachable is evidence of a different kind. Read the quoted packet inside it and identify which device emitted it, because a router's administrative rejection is not an authenticated error from the peer.

tshark -r ike.pcapng -Y "icmp.type == 3" -T fields -e frame.number -e ip.src -e icmp.code

Correlate the same attempt with server-side logs where you are allowed to read them. Did the request arrive, did the service answer, and where did the answer stop? Those three answers send you to a path problem or to a policy problem, and they cost less than rotating credentials a fourth time.