IPsec ESP: What It Carries and How to Prove It Is Flowing

ESP carries the traffic in an IPsec VPN, and a runbook for it starts with one rule: read the security association before you change a policy. Most unexplained ESP behaviour traces back to a selector that did not match, a mode that was assumed rather than read, or a counter nobody checked while a single flow was being sent. The three steps below establish what is carrying what, prove that one flow uses it, and leave a rollback that does not depend on memory. Each step produces one observation worth writing down.

Know the fields before you read them

ESP puts a header in front of the protected data and a trailer behind it. The header carries a security parameter index that identifies which association the receiver should use, a sequence number for replay detection, and padding in the trailer that keeps the payload aligned for the cipher. The next-header field in the trailer says what the protected content is, which is how the receiver knows what it decapsulated.

Mode changes what sits inside that protection. In tunnel mode the protected material is a complete inner IP packet, and a new outer header carries it between the endpoints. In transport mode the protection wraps the upper-layer payload of an existing packet, so there is no second IP header to unwrap. Which mode is in use is a configuration fact, not a matter of interpretation. Read it from the state entry instead of inferring it from a diagram.

Neither mode hides the outer addressing that has to deliver the packet. That is a property of the design rather than a shortcoming, and it is why an intermediate observer can still see which two addresses are talking even when the payload is opaque.

Step one: read the state, then the policy

The state entries hold the keys and counters; the policy entries hold the selectors that decide what may use them. On Linux both are visible without a packet capture, and reading them in that order keeps the two questions separate.

ip xfrm state
ip xfrm policy

In the state output, check the source and destination of the association, the mode, and whether the encapsulation shows UDP ports. Those ports mean NAT traversal is in use, which changes what an intermediate firewall must permit but not which selectors or keys apply at the endpoints.

Then read the selectors in the policy. A selector that names a narrower prefix than the traffic you are testing is the most common reason a working tunnel appears to ignore a flow, and a selector for the wrong protocol produces the same silence.

Vendor tools print the same information in a friendlier shape. ipsec statusall on strongSwan lists the child associations, their selectors and their byte counters, which is often faster than reading raw xfrm output when you already know which child to look at.

Step two: prove one flow moves the counters

Counters are only meaningful if you know what should have moved them, so pick a source and destination that the selector definitely covers and send one harmless request. Then read the same entry again and compare.

ip -s xfrm state
tcpdump -ni any esp

Two readings tell different stories. Packets that increment on the outbound side but never on the inbound side point at a forwarding or return-path problem rather than a key problem, because the packet was encapsulated and left. Neither counter moving means the traffic never matched the policy at all, and the selector is where to look.

The capture in the same step confirms the encapsulation on the wire, including whether the outer packets are ESP or UDP-encapsulated ESP. It also shows you the addresses that actually carried the flow, which occasionally differ from the ones in the configuration you were reading.

Step three: change one thing and keep a rollback

Record the association's source, destination, mode, selectors and counters before editing anything. If a change makes things worse, that record is what lets you restore the previous state rather than rebuilding a working configuration from a guess. That record takes a minute to write and saves an hour later. It is also the only artefact that survives the person who made the change.

Change one selector or one parameter, reload it, and repeat step two against the same flow. Deleting an association tears down every flow using it, so a targeted edit and a reload of the child is almost always better than flushing state to see what happens.

Rollback for a selector edit is restoring the previous policy and reloading, with the caveat that the peer may need a matching change if the selector is negotiated rather than local. Write that dependency down while it is fresh, because a half-restored pair is harder to reason about than either end state.

Finally, keep the scope honest. A flowing ESP association proves the two gateways are protecting and delivering the segment, and it says nothing about whether the application on top authenticates its users or encrypts what it stores. Those are separate controls with separate evidence, and the runbook is only finished when the next person can repeat it from the notes.