Docker Swarm Over VPN: Tasks Run, Service Calls Fail

Three managers and five workers across two sites, joined by a tunnel, and docker service ls shows tasks running on both sides while calls between services time out. Nothing in the Swarm logs looks wrong. That combination is the signature of a data-plane problem: the control plane found the nodes, and the encapsulated frames never made the trip.

The mesh has to exist before the overlay does

Swarm's overlay pushes encapsulated frames directly between node addresses, which means every node needs an address every other node can reach without translation, on the interface the tunnel provides. A tunnel that only carries clients to a published port is not enough. The underlay has to route node to node in both directions.

Test that before you touch Swarm. From each node, check the route to a peer's tunnel address, ping it, and then test the control port. If ping works and the port does not, the tunnel is fine and a filter is in the way.

ip -brief address
ip route get 10.20.0.12
ping -c3 10.20.0.12
nc -vz 10.20.0.12 2377

If the route points at the physical interface instead of tun0 or wg0, the overlay will encapsulate onto an address the far site cannot reach, and the symptom will read like packet loss.

Check what address each node advertises

The daemon advertises one address, and peers use it for the data plane. On a multi-homed host it is easy to advertise the LAN or public address while the tunnel carries something else. Inspect it rather than assuming, and pin it at join time with --advertise-addr when the default is wrong. A node that advertises an address only it can reach still joins the swarm, and the failure shows up later as services that cannot talk across sites.

Changing the advertised address later is not a config edit, because the node has to leave and rejoin.

Record the advertised address per node in your notes, since it explains most asymmetric failures.

Masquerade is the usual culprit

When a gateway sits between the nodes and the tunnel, the common mistake is to masquerade everything leaving the physical interface. That rewrites the overlay's inner addresses too, so every peer appears to come from the gateway and VXLAN peers stop recognizing each other. The result is missing or one-way encapsulated traffic while plain pings still work.

Route the overlay prefix through the gateway and leave it untranslated. If translation is genuinely needed for another prefix, scope it there and never to the range the overlay uses.

ip route add 172.16.0.0/22 via 10.20.0.1
iptables -t nat -I POSTROUTING -o tun0 -s 172.16.0.0/22 -j ACCEPT

Check the rule order as well as the rule text, because a broad MASQUERADE earlier in the chain wins over a narrow exception later.

VXLAN needs room in the packet

VXLAN adds 50 bytes to the frames it carries. A 1500-byte underlay carrying a 1500-byte inner frame produces an encapsulated packet that no path will deliver, and the failure looks like intermittent service discovery rather than a size problem. Either lower the tunnel MTU at both ends or set the overlay MTU so the driver advertises a payload that fits.

docker network create --driver overlay --attachable --opt com.docker.network.driver.mtu=1400 swarm-overlay

The ingress network is created automatically, so changing it means removing and recreating it rather than editing it in place.

If you only fix one side of the tunnel, the smaller MTU still has to be the one that matters.

Watch the data plane, not the logs

Capture the VXLAN port on both ends. Frames leaving one side and arriving on the other mean the underlay is fine and the problem is inside the overlay. Frames leaving and never arriving mean a route or a filter. Frames that never leave at all mean the source address is wrong.

Service discovery failures that clear up after a few seconds usually point at the packet budget or a flapping path, not at raft quorum.

Keep the capture from a working window, because comparing it with the broken one is faster than re-reading configuration.