AKS Can't Pull From a Private Registry Behind Your VPN
The kubelet pulls through the node network, not a pod's VPN. Decide by node routes, node DNS, every endpoint the pull touches, then TLS and identity.
Your laptop pulls the image fine, the pod sits in ImagePullBackOff, and somebody suggests running the corporate VPN client as a sidecar. That will not help, because the pull happens before your container exists: the kubelet asks the container runtime on the node to fetch the image, and the runtime uses the node's network namespace. A VPN inside the workload has no route to offer that path, and its own helper image would still need pulling first.
The order of operations decides the layer
Image resolution comes first. The kubelet, through the runtime, resolves the registry name, connects to it, authenticates, downloads layers, and only then starts the container. Everything your pod does with a VPN happens after that sequence completes, which is why a working in-cluster tunnel is not evidence about the pull.
This is also why a cached image is such a convincing liar. One node that pulled the image weeks ago keeps serving it, the deployment looks healthy, and the failure only appears when the scheduler places a replica on a newer node. If the event timeline includes a scale-up, a node replacement or a pool upgrade, treat those as the trigger rather than the cause.
Three conditions decide the repair
Private registry reachability from a managed node pool comes down to routing, name resolution and endpoint set. If the node subnet cannot route to the registry network, no credential will help. If it routes but cannot resolve the registry name from the node's DNS path, the connection never starts. If both work for the manifest endpoint but not for the blob endpoint the registry redirects to, the pull fails halfway with a misleading message.
| Condition | Typical symptom | Where to look |
|---|---|---|
| Node subnet has no route to the registry network | Timeout, no connection | Effective routes on the node NIC, gateway transit, return path on-premises |
| Node resolver cannot see the private zone | Name resolution failure in events | VNet DNS settings and the resolver the node actually uses |
| Only the manifest endpoint is reachable | Auth succeeds, layer download fails | Every hostname the registry hands out during the pull |
| TLS or identity is wrong | Certificate error, 401 or 403 | Server certificate name and trust, scoped pull identity |
VNet peering alone does not automatically give you every gateway-transit arrangement, and a route that exists in one direction is not a route that works. The node needs a supported path through the intended VPN gateway or approved appliance, and the registry side needs a return route to the node subnet.
Read the failure before choosing a fix
Start with the pod events and the node it landed on, using your real names: kubectl describe pod app-example -n team-example and kubectl get pod app-example -n team-example -o wide. The event text separates an unresolved hostname from a timeout, a certificate rejection from a missing image, and an authorization failure from all of them. Those four have four different owners.
Authentication is the most over-estimated lever. An imagePullSecret gives the runtime credentials for the registry; it does not install a route, change the node's resolver or make an untrusted certificate valid. If the event says the pull was unauthorized, look at identity scope. If it says the connection timed out, credentials are irrelevant.
The Kubernetes image documentation is the reference for pull behaviour and private-registry credentials, and it is worth reading once before you change networking, because it makes the boundary between authentication and transport explicit.
Build the path below the pod
Work in this order, because each step is meaningless until the previous one holds. Route the node network first: confirm the node subnet has a supported path to the registry network, check effective routes on the node, the security rules on both ends, and the return route on the registry side.
Then resolve the name from the node's context. The node's resolver is not the same thing as a pod's DNS configuration, and success inside a pod is not evidence about the host runtime. Use managed private DNS rather than a permanent hosts-file entry; the entry survives until the node is replaced and then silently disappears.
Then prove every endpoint. A registry can hand the client a separate authentication service and a separate blob storage location. Reaching the manifest endpoint proves almost nothing about the layers. Finally, complete TLS and authorization: a certificate valid for the registry name and trusted through supported node configuration, plus a pull identity scoped to read that repository.
Verify it on a node you did not repair by hand
The test that counts is a fresh pull and a full layer download on an eligible node, ideally one created by scaling or replacement. That covers every pool that could schedule the workload, including the ones you forgot. A manual fix on one node is not a deployment model, and the next autoscale event will find the gap.
Recreating the cluster in a different VNet can change the path, but it is the last resort rather than the first diagnostic step. Compare the existing node network against the required private route and the supported AKS networking controls before proposing a migration, and keep application connectivity after startup as a separate verification: the pod talking to its backend is a different check from the node downloading the image.