VPN client IP inside Kubernetes: preserve it or SNAT it?

Somewhere between the tunnel and the backend, a client address either survives or gets replaced, and both outcomes are defensible. The choice belongs to whoever owns the receiving service, not to whoever installed the CNI. When per-client authorization, quotas or audit records matter, the address has to survive every hop. When shared attribution is acceptable, source NAT removes a routing problem you would otherwise maintain forever.

Where the client address actually disappears

A remote client's packet arrives at the VPN pod with an inner source inside the tunnel range, say 10.90.0.42. When the pod forwards that packet toward a ClusterIP or a pod IP, three different mechanisms can rewrite the source. The node's CNI masquerade rule covers traffic the pod sends outside the cluster CIDR. kube-proxy's conntrack entry on the destination node covers the Service hop itself. A proxy that terminates the connection does not forward the packet at all, so the source you eventually see belongs to the proxy.

Each lives in a different layer, which is why one Service field almost never repairs the whole path.

externalTrafficPolicy on the VPN's public Service decides only what happens to the outer connection: Local keeps the remote address on the node that received it, while Cluster lets kube-proxy hand the packet to another node and masquerade it there. That setting governs the listener your clients dial. It says nothing about the decrypted packet your VPN process routes into the cluster afterward.

Three conditions that settle the argument

I ask the same three questions in the same order every time this comes up, because the first one usually ends the discussion. Answer them for the specific workload rather than for the cluster as a whole. Two clusters inside one company can land on different answers and both be right.

If the backend answers yes to the first question and no to the second, you have network work to do rather than a manifest to edit. If the third row says proxy, the address was never going to survive, and pretending otherwise costs a week of packet captures.

What preserve mode demands from the network

Preserving the source means the destination node must return traffic to 10.90.0.42 through an active VPN gateway. The VPN pool is not a cluster CIDR, so no CNI routes it for you by default. You add the route yourself, then make sure the gateway endpoint it points at survives a reschedule.

ip route add 10.90.0.0/24 via 10.244.3.17 dev eth0
sysctl -w net.ipv4.conf.eth0.rp_filter=2

That next hop is a pod address, and pod addresses move. Point the route at a Service VIP or a node address that stays put, or accept that you will chase the gateway after every eviction. Reverse path filtering is the other trap: strict mode drops packets whose return path does not match the interface they arrived on, which is exactly the shape of a preserved VPN flow.

Loose validation is often the difference between a working return path and replies that vanish after leaving the backend. Masquerade rules on the VPN node also have to exclude the tunnel range, or the forwarded packet leaves with the pod's address no matter what route you built. Prefer expressing that exclusion in the CNI's own configuration instead of editing iptables by hand, because the agent rewrites its chains on restart and your hand-written rule survives exactly until the next upgrade.

NetworkPolicy is access control, not routing. Tightening a policy will never restore an address that masquerade already replaced.

What SNAT costs, and how to write it down

Source NAT toward the cluster is the cheaper design, and it is not a mistake. The backend sees the VPN pod or node address, logs agree with each other, and no return route has to exist. What you lose is the ability to tell two clients apart at the network layer, so quotas, bans and audit trails have to come from somewhere else.

Usually that somewhere else is the application: a session token, a signed header, or a PROXY protocol line written by a proxy you control. Those give the receiver an identity it can trust only when the proxy sits in the path and the header cannot be set from the client side. Be explicit in the runbook about which one applies, because the next engineer will assume the address in the log is the client. It is not, and one incident is enough to make that assumption expensive.

The trade stays honest as long as it is written down.

Checks that separate a working path from a lucky one

Test with a fresh connection rather than a cached one, and test in both directions. Several failures look intermittent only because new flows take the rewritten path.

Capture the tuple at three points during one authorized request: right after decapsulation, at the VPN pod's egress interface, and on the backend. If the source changes between the second and third capture, the cause sits downstream of the pod. If it changes before the second, the pod itself is translating.

Then delete the VPN pod and repeat the test while it reschedules. A preserved path that breaks on eviction is unfinished, and so is a NAT rule pinned to an address that no longer exists. Finally, send one request from a source you expect to be denied and confirm the refusal comes from policy rather than from a missing route.