How to Read a Fresh IKE_SA_INIT Exchange in a Capture

Start from a capture that begins before the first packet of the attempt, or you will be reading the middle of a conversation. A negotiation that is already running shows encrypted exchanges and rekeys, and neither looks like the opening. The opening is the part you can decode without keys, which is why it answers reachability questions so well. Four checks in order take you from a fresh capture to a defensible statement about which side stopped talking.

First, Confirm the Capture Starts at the Beginning

A capture started after the tunnel came up will contain the encapsulated data stream and periodic keepalives, and both are either encrypted or nearly empty. Restart the client and record the attempt itself, or trigger a fresh connection deliberately while the capture runs. If you cannot produce a fresh attempt, say so in your notes, because no amount of decoding recovers an opening you never recorded.

Scope the capture to the two endpoints and to UDP 500 and 4500 so the file stays small enough to read. On a busy host an unfiltered capture buries the negotiation under unrelated traffic.

tcpdump -i any -n -s 0 -w fresh-ike.pcap 'udp port 500 or udp port 4500'

Keep the file when the attempt is interesting, because the same capture answers both of the questions that follow: what the exchange was, and whether anything came back. Label it with the start time and the endpoint pair you filtered on. A capture without those two facts is useless a week later, when addresses have changed and nobody recalls which attempt it recorded.

Second, Read the Header Before the Payloads

The IKE header carries the security parameter indexes, the exchange type and the message ID, and on a fresh negotiation those values are distinctive. The initiator invents a nonzero SPI, the initial request carries a responder SPI of zero because no reply has been seen yet, the exchange type is IKE_SA_INIT, and the message ID is zero. Anything else belongs to a later stage of the same negotiation or to a different attempt entirely.

The proposal, key exchange and nonce payloads in this first exchange are readable in the clear. Readable is not authenticated, and at this point neither peer has proved its identity.

The private Diffie-Hellman value never crosses the wire, so the capture does not let you derive session keys. Authentication happens in the next exchange, protected by what the first one negotiated. If someone asks whether a peer is genuine, this exchange cannot answer that question. It can only show that both sides agreed on a proposal and produced a shared secret the file does not contain.

Third, Match the Request to Its Response

Match on the SPI pair, the exchange type and the message ID, then check the response flag. A retransmission repeats the message ID of the original and is not a second login attempt. Counting packets instead of exchanges is how a healthy negotiation turns into a phantom loop.

A reply can also be a refusal: a cookie request, an unacceptable group notification, or an error about a proposal. Those are legitimate answers, and they mean the path works while the policy does not. Decode the notification instead of treating every reply as success. When the notification names a group or a proposal, the repair is on the configuration side rather than at the firewall.

Direction matters too. A request with no response and a response that never reaches the client are different faults, and only the second one looks like a client-side timeout.

Fourth, Separate IKE From Everything Else on 4500

Port 4500 carries three things that look alike in a summary view: IKE behind a four-byte zero non-ESP marker, encapsulated ESP whose marker position holds a nonzero SPI, and short keepalives. That marker is what separates IKE from encapsulated ESP, so read those bytes before you label the traffic IKE. A capture label reading 4500 is a port number, not a protocol.

Encapsulated ESP is encrypted payload you cannot interpret without keys, and its presence may be the sign of a working tunnel rather than a fault.

Watch both ports for the same attempt, because IKE can move to 4500 after NAT detection even though it opened on 500. A trace filtered only on 500 looks silent exactly when the negotiation succeeded and relocated. Widen the filter and re-read the same time window before drawing any conclusion from absence.

What Silence at One Observation Point Means

No reply at the client proves only what that point observed. Filtering, routing, the wrong remote address, a service that is down and a protocol mismatch all produce the same blank trace.

A relevant ICMP unreachable adds evidence, but read the quoted packet inside it and note the address of the device that sent it.

A router's rejection is a different event from an authenticated error returned by the peer, and only the second proves the service is alive. Correlate the same attempt with server-side logs or an approved observation to learn whether the request arrived and whether a response left. Keep captures scoped and private, since readable handshake fields do not make the rest of the file harmless. The result worth having is one matched request and response, or one documented gap with a named observation point.