GKE Pods Lost VPN Access After 1.14: Two Ways to Fix the Source

An upgrade lands, and systems across the tunnel begin rejecting pod traffic that worked the previous week. The tunnel is up, the firewall rules are untouched, and nobody edited the peer policy. What changed is invisible in the cluster logs: the source address those systems now see. That gap explains a failure which looks like a network outage and behaves like a permissions problem.

Two valid answers exist, and they send you to different places. Choose the source contract before editing a ConfigMap.

The default rule that stopped being installed

Earlier GKE releases installed a masquerade rule that replaced pod sources with the node address whenever the destination sat outside 10.0.0.0/8. A VPN gateway inside that range saw the pod address, while anything further away looked like it came from the node. Peers built allowlists and return routes around node addresses because that is what they observed.

Later releases stopped installing that default rule. Pod traffic now arrives at the gateway with pod addresses, whatever the destination is. A peer that only routes node addresses drops the replies, and neither side writes a useful log entry. The request may even succeed on the far end, which is what makes the failure confusing to chase.

Nothing is broken in the sense of a dead component. The two sides simply disagree about which source is legitimate.

Choose the source the peer should see

Keeping pod addresses is the more accurate contract. The far side sees the workload that produced the traffic, logging stays honest, and per-pod policy becomes possible. The price is that the peer must carry the pod prefix.

Keeping node addresses is the cheaper contract. Turn masquerading back on for that destination and the peer keeps seeing an address it already trusts.

One question decides between them: what does the peer do with the source? If it logs, applies per-source policy or enforces narrow return routing, pod addresses justify the extra configuration. If it only needs to send replies, masquerading costs less and asks nothing of the other side. Either choice is defensible as long as both teams write it down.

A third option is usually a mistake: masquerade everything and lose the ability to attribute traffic. Avoid it unless a peer genuinely cannot route anything except node addresses.

Point nonMasqueradeCIDRs at the real ranges

GKE runs the ip-masq-agent in kube-system, and its ConfigMap lists the destinations that skip masquerading. To preserve pod sources toward the tunnel, the pod range, the node subnet and the on-premises prefix all belong in that list.

Replace the sample values with ranges from your own cluster. Keep the list no wider than the requirement dictates, because every extra entry changes what some other destination sees. The three entries below stand for the pod network, the node subnet and the prefix behind the tunnel:

nonMasqueradeCIDRs:
  - 10.12.0.0/14
  - 10.100.0.0/16
  - 172.16.0.0/12
masqLinkLocal: false
resyncInterval: 60s

An entry of 0.0.0.0/0 disables masquerading for all destinations, which alters how unrelated workloads appear to outside services and is rarely intended. The agent re-reads its configuration on resyncInterval, so a restart is usually unnecessary. Check that the agent pod is running, then open the mounted file inside it. Reading that file is what proves the change landed.

The reply path is the other half

Masquerading alters the request source and nothing else. Whatever source the peer receives, it still needs a route back to it, and the cluster still needs a route toward the peer.

A capture on the VPN gateway separates the two failures quickly. Packets that arrive with pod addresses and never get an answer mean the peer lacks a route for the pod prefix. An answer that appears on the gateway but never reaches the cluster means the local route is missing. Write the contract into the ticket either way: which source the peer sees, and which prefix it routes back.

Re-check after every upgrade

Default masquerade behaviour has changed once already, and node pool replacement can reset cluster networking. Treat the source contract as something to verify rather than something to assume.

After an upgrade, send one request from a pod to an on-premises service and read the source the peer logs. That single observation catches a regression which a tunnel status page will happily report as healthy. Repeating it costs a minute and saves the next person an evening.