kubectl can't resolve private cluster API over OpenVPN

kubectl fails at name lookup while the tunnel is up and the API's private address still answers a ping. The route is fine; the client is asking a resolver that has never heard of the cluster's private zone. Two fixes get used most: push the virtual network's resolver to every VPN client, or pin the name locally with a hosts entry or a conditional forwarder. They solve the same symptom with different ownership, and mixing them up is how teams end up editing hosts files every time an endpoint is rebuilt.

Resolution and reachability fail at different layers

Run the lookup yourself before changing anything. getent hosts on Linux or nslookup on Windows tells you whether the client gets an answer at all; a timeout or NXDOMAIN points at the resolver, while an answer with the wrong address points at a stale record or a public fallthrough.

Then confirm which server kubectl actually dials. A kubeconfig context stores a URL, and a VPN route does not rewrite it: if the context names a public endpoint, DNS can be perfect and the command still fails.

getent hosts api.internal.example
kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}'
kubectl -v=8 get nodes 2>&1 | head -40

The third signal is the shape of the failure after the lookup. A name that resolves to the private address followed by a TCP timeout is a firewall or authorized-network problem. A name that resolves to a public address means the query escaped the private zone. Keep those three outcomes separate; each one sends you to a different owner.

Option A: push the virtual network resolver into the tunnel

When the VPN server runs inside the virtual network, clients can use the same resolver as the server host. In OpenVPN Community the server pushes a DNS option; Access Server exposes the same setting in the admin UI, and clients must download the profile again because the DNS directive travels inside it.

# server.conf
push "dhcp-option DNS 10.0.0.4"
push "route 10.0.0.4 255.255.255.255"

The route line matters on split-tunnel profiles. Pushing a resolver address is not the same as making it reachable: if only the cluster subnet goes through the tunnel, the query to 10.0.0.4 leaves via the local LAN and dies. Push a host route for the resolver, or include its subnet in the tunnel.

The receiving side also has a policy. A resolver reachable from the client address range can still refuse queries from it, and a private zone linked to the virtual network answers only clients that arrive through that network's DNS path. If the lookup times out after the push, check the resolver's own logs before touching the client again.

On the client, verify what was applied: resolvectl status on systemd-resolved Linux, scutil --dns on macOS, ipconfig /all on Windows. A pushed DNS server that never appears means the profile was not reloaded.

Option B: hosts entries and conditional forwarding

A hosts entry is the smallest possible fix and the most brittle. It maps one name to one address on one machine, so it works for a demo and breaks when the private endpoint or its IP is rebuilt. It also hides the real gap from everyone else on the team.

A conditional forwarder scales better without pushing DNS to every client. dnsmasq can send one zone to the VNet resolver and leave everything else alone: server=/cluster.internal/10.0.0.4. Windows DNS has conditional forwarders, and Windows clients can use an NRPT rule for the same split. The forwarder still needs a route to the VNet resolver.

Forwarding keeps normal browsing local, which matters on split tunnel. It also gives you one place to log queries for the private zone, and one place to break when the forwarder host moves.

Pick by ownership, not by convenience

Push the resolver when you control the VPN profile and want every private name to work, not just the API. Choose a forwarder when clients must keep their local DNS for everything else, or when the VPN is managed by a different team than the cluster.

Hosts entries are a diagnostic, not a deployment. Use one to prove the API is reachable, then remove it and fix the resolver path; leaving it in place turns a routing question into a scavenger hunt later. Whichever option you pick, write down the zone, the resolver address and the route that carries the query.

Test the API endpoint, not just the name

Once the lookup succeeds, fetch the API's version endpoint with the cluster CA. curl --cacert ca.crt https://api.internal.example:6443/version separates TLS and authorization failures from DNS. A 403 means the name and connection work and the identity does not.

Also check whether the client address range is allowed by the cluster's authorized networks or firewall. Private clusters often accept API traffic only from the VNet or peered ranges, so a tunnel that assigns addresses from a different pool connects at the network layer and is dropped at the API.

Record four facts when you escalate: the resolver the client used, the address it returned, the server URL in the selected kubeconfig context, and the first operation that failed. That record tells the next engineer whether to change DNS, routing or firewall policy, and it stops the next hosts-file edit.

Start with the lookup, choose the resolver path that matches who owns the zone, then verify the endpoint. If all three pass, the remaining problem is authorization, not name resolution.