Why GKE Pods Cannot Reach On-Prem Hosts Over VPN

Ping works from the node and fails from the pod, and nothing arrives on-premises. The tunnel and the route are fine; the source address on the wire is not. What differs between those two cases is what the packet claims to be when it leaves the cluster, and GKE has an opinion about that. The mechanism is a masquerade exclusion list, and reading it takes one command.

Read the effective list before changing anything

GKE ships an agent that decides, per destination range, whether pod traffic gets rewritten to the node address on the way out. Its shipped configuration names the three RFC1918 blocks, 10.0.0.0/8, 172.16.0.0/12 and 192.168.0.0/16, as destinations that are exempt. Anything you send from a pod to a 10.x host on-premises therefore leaves the node with the pod's own address still attached. The node's own traffic is not rewritten, which is exactly why its ping works.

Read the intended configuration and the installed rules, because they can disagree. kubectl -n kube-system get configmap ip-masq-agent -o yaml shows what the agent was told, and kubectl -n kube-system get pods -l k8s-app=ip-masq-agent shows whether anything is running to apply it. On a node, iptables-save | grep -i masq shows the rules that actually exist in the kernel. When the ConfigMap says one thing and the rules say another, the agent is the suspect rather than the network.

Confirm the source address with a capture

Capture before you change a list. A filter on the one destination keeps the output readable: tcpdump -ni any host 10.20.30.40. A line that prints the pod address as the source, with no translation, means the exclusion list is doing what it was configured to do.

Compare that with what the far side reports. A capture on-premises that shows the pod address and no reply closes the loop: the exclusion list explains the asymmetry without any tunnel fault. Two observations of one source address are enough to stop guessing about the CNI.

Narrow the exclusion to the pod CIDR

The supported repair narrows the exemption instead of removing it: keep the pod CIDR exempt and drop the rest of the 10.x space. That list lives in the agent's ConfigMap under nonMasqueradeCIDRs, and changing it is an edit followed by a rollout. Replace the wide entry with the cluster's actual pod range, keep the other private blocks if in-cluster traffic still needs them, and leave resyncInterval alone unless you have a reason. Apply it, confirm the agent pods restarted, and confirm the kernel rules changed.

This is a cluster-wide change with a real blast radius. After it lands, traffic that used to keep its pod source now arrives rewritten to the node, so anything that keyed on those pod addresses sees a different sender. The casualties are logs, allowlists and on-premises firewall rules that name pod addresses, and they fail quietly. Announce the change as a behaviour change rather than a fix.

On a cluster with several node pools, check that the agent is scheduled on the nodes where the affected pods actually run. A pool without the DaemonSet keeps its old rules, and nothing in the ConfigMap will show that.

The alternative and its cost

The other option keeps pod addresses intact and teaches the path about them. The pod CIDR goes into the Cloud VPN traffic selectors, and the on-premises side needs a return route for it so replies know where to go. That preserves pod identity, which some security models want, and it broadens both the routing and the firewall scope the far side has to accept.

The trade is legibility against scope. Translation makes the on-premises side see one address per node, which is easy to allowlist and hides which pod sent what. No translation keeps the pod address end to end, which is better for attribution and worse for every device in the path that has to learn a new range. Neither option is a Docker or CNI defect; both are routing decisions with an operational bill. Pick the one whose bill your team can pay, and write down why.

Verify and roll back

Run the same capture again. A node-level capture that still prints the pod address after your edit means the rules never changed, whatever the ConfigMap says. The wire is the fact; everything else is inference.

Then check the reverse direction, because half of these changes get verified only outbound. Confirm the reply reaches the node or pod that sent the request, and confirm the application completed rather than merely receiving a SYN. A capture showing a request with no reply is a routing problem on the far side, not a masquerade problem on this one.

Rollback is the previous ConfigMap, applied the same way, with the same confirmation that the agent restarted and the rules changed. Keep the before and after source address for one known request in the change record, since that single line is what proves the change did what you said. The GKE IP masquerade agent guide documents the defaults and the ConfigMap fields. If the source address is unchanged after the rollout, the agent is not running on that node and no amount of YAML editing will fix it.