AH or ESP? Decide by what the packet has to protect

Two IPsec proposals can name the same SHA-based integrity algorithm and still ask for different protection protocols; AH and ESP are not two spellings of one authentication setting. Read the protocol first, then its algorithms, then its mode, and most of the ambiguity disappears. AH authenticates the packet including the IP header fields that survive transit, while ESP authenticates and encrypts what it encapsulates and leaves the outer header alone. That difference decides more deployments than any hash name does.

What AH covers that ESP does not

AH provides integrity, data-origin authentication and the anti-replay service it defines, and no confidentiality at all. Its coverage includes upper-layer data plus the IP header fields its rules treat as immutable, which is exactly the part ESP in transport mode leaves outside. When a requirement says the IP header itself must be authenticated, AH is the protocol that answers it.

That coverage is also why AH is fragile in the field. Mutable fields such as TTL, the header checksum and some options cannot be authenticated because routers may rewrite them, so the protocol zeroes them before computing the check. Anything that rewrites addresses therefore breaks the calculation rather than merely the policy.

NAT turns the choice into a constraint

Address translation rewrites the source or destination in the IP header that AH has just authenticated, so the integrity check fails at the far end, and it fails as a protocol error rather than a message you can read. ESP does not authenticate the outer header, which is why ESP with NAT traversal works and AH behind NAT does not. A tunnel whose peers sit behind ordinary routers is exactly where this appears first. The first question is therefore not which protocol is stronger but whether a translator sits between the peers.

If one does, AH is out, and the conversation moves to ESP and its algorithm choices. That is a constraint rather than a preference, which is why it belongs at the top of the decision instead of the end.

Mode multiplies the question

ESP in transport mode protects the upper-layer payload and leaves the IP header alone; in tunnel mode it encapsulates the whole inner packet, including the inner IP header, and protects that. AH has the same two modes, so AH or ESP is really four combinations before any algorithm is chosen.

Combined-mode ESP algorithms then settle integrity without a second protocol. AES-GCM and ChaCha20-Poly1305 authenticate and encrypt in one construction, so adding AH on top buys nothing while costing another header and another negotiation. The architecture permits AH and ESP together, and the practical question is whether your gateway and release support the combination and whether anyone can maintain it afterwards.

The honest replacement for AH-only

The integrity-only ESP variant keeps the algorithm and drops the encryption claim instead of faking it, and it survives NAT. If a legacy design used AH because it needed authentication without confidentiality, that variant is the direct translation rather than an encrypted tunnel with a note attached.

Seeing ESP in a proposal is therefore not evidence of encryption either. Read the encryption algorithm: a NULL encryption entry means integrity only, and a combined-mode entry means both.

Decide in three questions

Ask whether confidentiality is required, whether a translator is on the path, and which header fields must be authenticated. Those three answers point at one protocol and one mode, and writing them down takes less time than reading the RFCs again later.

Then verify what was accepted rather than what was proposed. ip xfrm state shows the installed association's protocol, mode and algorithms, and a mismatch leaves nothing installed at all, which is a cleaner failure than a tunnel that comes up without the protection you intended.

Keep authentication failures, selector coverage and routing faults separate from this choice. They produce different symptoms, and blaming the protocol for a selector problem is a dependable way to spend a day changing hashes for no reason.