Transport or tunnel mode: decide by the packet, not the diagram

Textbook diagrams pair transport mode with two hosts and tunnel mode with two gateways, and that rule of thumb survives right up to the first deployment that breaks it. Mode describes what the ESP packet wraps, not who the peers are. A gateway can protect traffic in transport mode, and a pair of hosts can use tunnel mode if they choose to. Three questions settle what a specific deployment needs.

Mode is a layout, not a topology

In transport mode the ESP header goes after the IP header and before the next-layer payload, so the original IP header stays outside and visible. In tunnel mode the protected content is a complete IP packet, and a new outer header names the two IPsec peers.

With confidentiality selected, the protected region in tunnel mode includes that inner IP header, which is why a capture cannot tell you the real endpoints without keys. The outer header stays readable either way, and that is what makes troubleshooting possible at all.

Question one: what exactly is inside the ESP

Look at the packet the application produces and ask which boundary protects it. If the answer is the transport payload between two addresses that are already correct, transport mode fits without inventing a second header. If the protected content needs its own addresses, because the real endpoints differ from the peers carrying the traffic, tunnel mode does the job. Both cases exist in the same organisation, which is why one policy statement rarely covers everything.

That question is answerable from a capture and a routing table. It is not answerable from a drawing that labels boxes as client and server. A capture taken at either peer shows which of the two you are looking at within seconds.

Question two: is a tunnel already inside

L2TP is the historical counterexample that makes the point. The service carries PPP inside an L2TP session, and IPsec protects that exchange; the arrangement specifies transport mode, with tunnel mode permitted as an option. The visible result is a tunnel at the service level whose outer protection uses transport mode.

So the presence of the word tunnel in a product description says nothing about the mode of the layer beneath it. Record which question settled it, because the next reviewer will ask the same thing. Read the specification for that combination instead of reasoning from the name, since that habit prevents most arguments about what a product must support.

Question three: what does the peer require

Two peers have to agree, and a proposal that offers one mode while the other requires the second fails during negotiation rather than degrading quietly. Ask the owner of the far end which mode the policy names, and check the proposal rather than the documentation summary.

Mixed estates are common: one gateway terminates transport-mode sessions for a legacy product and tunnel-mode sessions for a site link. The mode is per association, not per device. Ask for the association listing rather than the feature name, since the name travels faster than the configuration does.

Reading the association that exists

For a review, list the peers, the mode, the selectors and the next-layer protocol, then add any encapsulation wrapped around the result. Address translation adds its own outer view and is not a mode; a translated flow can still be either one. The listing is the only artefact that shows the association as installed rather than as intended.

Finish with the routing and authorisation question, because a correct packet layout tells you nothing about which subnets are allowed or whether the reply path exists. Those live in policy and in the route table, and neither one changes the argument about mode.