How to Split DNS Between Kubernetes and an OpenVPN Sidecar

Pods that run an OpenVPN sidecar fail name resolution in a recognisable pattern: cluster names hang, remote names answer until the pod restarts, and your edited resolver file reverts. Nothing in that sequence is a tunnel fault. A pod has one resolver file and two naming worlds, and the file can list servers in order but cannot express zones. The durable fix is a resolver that runs inside the pod and owns the split, so the file stays trivial and no policy has a reason to rewrite it.

Why the edit you made keeps disappearing

The file belongs to the kubelet, not to you. Under dnsPolicy: ClusterFirst the kubelet generates it to point at cluster DNS, and regenerates it whenever a container starts. Whatever you change through kubectl exec lives in the container's writable layer, so a restart, an eviction or a rollout discards it. The pod spec still carries the same policy, which is why the same file comes back.

Setting dnsPolicy: None stops the regeneration and also removes cluster resolution. You gain a file you control and lose the names you needed. That trade is the whole problem: resolv.conf has no notion of zones, only of servers to try in order, so it cannot send cluster.local to one address and a private zone to another.

Run one resolver in the pod, on loopback

Point the pod at a single nameserver on loopback, then let a CoreDNS container sharing the pod network namespace answer on port 53. Set the policy to None, hand the pod that one address, and add the DNS container beside the VPN and the application:

spec:
  dnsPolicy: None
  dnsConfig:
    nameservers:
      - 127.0.0.1
    options:
      - name: ndots
        value: "5"
  containers:
    - name: dns
      image: coredns/coredns
      args: ["-conf", "/etc/coredns/Corefile"]
      volumeMounts:
        - {name: corefile, mountPath: /etc/coredns, readOnly: true}
    - name: vpn
      image: your-openvpn-client-image
    - name: app
      image: your-app-image
  volumes:
    - name: corefile
      configMap:
        name: split-dns-corefile

All three containers share one network namespace, so 127.0.0.1:53 from the app container reaches CoreDNS with no service, no port-forward and no detour through the cluster network. Mounting the Corefile from a ConfigMap keeps it a versioned input instead of a file edited in place. The generated resolv.conf then holds one nameserver and nothing else, which is a state you can reason about.

Two details bite here. The official image binds port 53 as root, and some cluster policies make you spell that out. Every container in the pod now depends on that one container being up, so add a readiness probe against a harmless name and let the app tolerate startup failures.

Give each private zone its own upstream

A Corefile is a list of zones, each naming the server that can answer it. One block covers the tunnel zone, and the catch-all comes last and takes everything no explicit block claimed. The catch-all must aim at a real upstream, never back at the pod: .:53 { forward . 10.100.0.10 } reuses the cluster resolver because that resolver already forwards public names to the node's own resolvers. Nothing else in the pod needs to know which network a name belongs to.

.:53 {
  errors
  cache 30
  forward . 10.100.0.10
}

svc.acme.internal:53 {
  errors
  cache 30
  forward . 192.168.240.1
}

Read the cluster address off the kube-dns service in kube-system rather than copying one from an article, because service CIDRs differ. The tunnel address comes from the dhcp-option DNS value the OpenVPN server pushes, or from the authoritative server for that zone. If your cluster resolver does not answer public names, put the node's resolver address in the catch-all instead, read on the node and not inside the pod.

Do not point the catch-all at /etc/resolv.conf, which is the tempting one-liner and a trap in this exact setup. Under dnsPolicy: None the pod's own file contains the single nameserver 127.0.0.1, so that target is CoreDNS itself and every unmatched query recurses until it times out. A block named svc.acme.internal matches that name and every name beneath it, which covers db.svc.acme.internal and excludes svc.acme.internal.example.net. The trailing dot is cosmetic, because zone names are normalised to fully qualified form; the suffix boundary is what decides where a query lands.

Decide whether the forward belongs in the cluster resolver

If every workload in the namespace needs the tunnel zone, a forward block in the CoreDNS ConfigMap under kube-system is cheaper than a sidecar per pod. That change is cluster-scoped: it needs administrator rights and a restart of a deployment everyone depends on. Scope it to that one zone, never to the root. A single workload's tunnel address then becomes every pod's resolver for the zone, and the day the tunnel drops the whole cluster loses the name.

The sidecar keeps the blast radius inside one pod, which is the better default when only a few services need the remote network. Whichever you choose, write down which resolver owns the zone, because the next person debugging a timeout will ask.

Names that resolve while connections still fail

Resolution is one of the two things the tunnel has to carry; the other is the return path, and pod networking hides it well. If the OpenVPN server itself runs as a pod, rules you add inside that container do not govern traffic that leaves through the node. That failure is narrow: only pods on the node that also ran the server got answers, and everything else timed out.

Running the server with hostNetwork: true puts it back in the node namespace, where a FORWARD rule and source translation take effect. Translate the client pool on the way out, or no remote node has a route back to a tunnel address such as 10.8.0.6. Check both directions before you touch the Corefile again. A tunnel that carries the query and not the reply produces exactly the same timeout as a resolver that never received the query.

Verify from a new pod, not the session you fixed

Ask both zones explicitly, then repeat the query after the pod is replaced. Configuration that exists only in a running container is not configuration:

kubectl exec -it deploy/app -- dig @127.0.0.1 kubernetes.default.svc.cluster.local +short
kubectl exec -it deploy/app -- dig @127.0.0.1 db.svc.acme.internal +short
kubectl delete pod -l app=your-app
kubectl wait --for=condition=Ready pod -l app=your-app --timeout=90s
kubectl exec -it deploy/app -- dig @127.0.0.1 db.svc.acme.internal +short

If the first pair answers and the pair after the replacement does not, the Corefile is fine and the pod spec is not. If both answer and the application still cannot connect, stop debugging DNS and test the TCP path with nc -vz db.svc.acme.internal 5432 from inside the container. Keep the dig output in the ticket, because it is the artefact that separates a naming problem from a routing problem.