Private GKE nodes do not decide which API kubectl uses

The nodes are private, the VPN reaches them, and kubectl still cannot talk to the cluster. Those two facts are unrelated. The address kubectl uses comes from the server URL stored in kubeconfig when the credentials were fetched, and no route you add changes it. Meanwhile the nodes are not what kubectl is trying to reach in the first place.

Five checks, starting with the only value that matters.

Check 1: read the server URL in kubeconfig

Print the endpoint without dumping the credentials: kubectl config view --minify -o jsonpath="{.clusters[0].cluster.server}". That one string tells you whether the client points at an IP address, a DNS name, or an external address you did not intend. Everything else in this investigation follows from it.

If the URL is not the endpoint you thought you configured, the rest of the diagnosis is a distraction.

Check 2: match the URL to the endpoint type

Current GKE supports both IP-based and DNS-based control-plane access, and the credentials you fetch select one. gcloud container clusters get-credentials NAME --location REGION --internal-ip writes the internal IP endpoint, while --dns-endpoint selects DNS-based access. Neither flag provisions a network path, which surprises people who expect the fetch to fix connectivity. Re-run the fetch deliberately and read the URL again afterwards.

Changing flags without re-reading the URL is how a laptop ends up holding two clusters of credentials and one confusing error.

An authorized-network entry governs who may reach an external endpoint and does not apply to the internal one. Widening it to test a private path changes your exposure rather than your diagnosis.

Check 3: prove the path to that endpoint

If the URL is an internal IP, verify the route from the network the client actually sits on, the allowed source range and the cluster regional or global access setting. The control-plane range and the node subnet are not the same destination, so a working route to the nodes proves nothing about the endpoint. For DNS-based access the path is Google API connectivity with its documented Private Google Access arrangement, not a route to the worker subnet. Test the endpoint with a plain TCP connection before blaming kubectl.

A timeout is a path problem. A 403 is a policy problem. They do not share a fix.

Check 4: separate logs and exec from API calls

If ordinary API requests succeed and kubectl logs or kubectl exec fails, you are already past endpoint selection. Those operations need the corresponding Kubernetes permissions and depend on control-plane-to-node communication, and the failure can be a Forbidden response rather than a timeout. SSH access to a node tests another service entirely and does not establish them.

Read the error class before reopening the VPN ticket, because the two classes live in different places.

Check 5: keep three observations for the next person

Record the selected API URL, whether one authorized API request works, and exactly which later operation fails, with the error text. Those three lines separate endpoint selection from node access, and they stop a perfectly good VPN route from being blamed for a request it never carried.

The next engineer will ask whether the tunnel is broken. Your note should already answer that.