Synology OpenVPN Handshake Fails? Check the LAN Forward

Your laptop reports that the TLS handshake failed, so the temptation is to rebuild certificates. Before that, ask a duller question: which Ethernet port is the Synology actually listening on? A NAS with two LAN ports holds two addresses, and DSM's VPN Server package binds its OpenVPN listener to one of them. When the router forwards 1194 to the other address, no packet ever reaches the service, and the client has nothing to negotiate with.

The error text misleads because it describes the last thing the client tried rather than the first thing that went wrong. Two minutes comparing the package page with the router rule settles it.

Find the address the listener is bound to

DSM labels its ports LAN 1 and LAN 2, and each carries a separate address and subnet. Open VPN Server, read the interface named on the OpenVPN tab, then open Control Panel and note the IPv4 address DSM has on that same interface. Those two values, not the NAS hostname, are what the forward must name.

A second cable plugged in does not create a second listener. Both links can show as connected while only one address accepts the traffic, and when both interfaces sit on the same subnet the two addresses differ by a digit that nobody re-reads before saving a rule.

Look at the socket itself over SSH, because this is the only place that shows what the daemon actually bound rather than what the settings page intends.

sudo netstat -anp | grep 1194

Edit the existing forward rather than adding another

Two rules for one external port leave the router free to pick, and which one it picks can change after a firmware update or a reboot. Open the rule that already exists, change its internal address, save, then reopen it and read the value back. Naming the rule after the service helps, because a list of five identical forwards is how the wrong one survives.

external 1194/udp  ->  192.0.2.20:1194/udp

Keep the internal port equal to the listening port, and keep the protocol consistent. UDP 1194 is the default listener, while a TCP profile needs its own listener on the package side and a matching forward on the router. Forwarding TCP to a service that only listens on UDP produces silence, not an error.

If the NAS address comes from DHCP, reserve it before the rule is saved. Lease expiry reassigns the address, the forward keeps pointing at the old value, and the handshake starts failing on a day when nobody touched the NAS. A static configuration on the NAS side solves the same problem, provided the address sits outside the router's DHCP range.

Prove whether packets arrive at all

Start a capture on the NAS first, then connect from the client, and read the source address in the output. A capture started after the failure shows nothing and proves nothing.

sudo tcpdump -i any -n udp port 1194

Silence means the traffic died before the service: a wrong internal address, a closed port on the DSM firewall, or an interface that is administratively down. Packets in the output mean the forward works, and the remaining fault lives above the network layer, which is a much smaller search area.

The capture also names the interface the packet arrived on, and that single detail separates a rule aimed at the wrong port from a rule aimed at a port that is not up. Watch the router's own counter during the same attempt if it exposes one. A counter that does not move while the capture stays empty points at the router, while a counter that moves with an empty capture points at the path between the two devices.

When packets arrive, stop blaming the forward

The profile is the next suspect, and the usual reason is age. An exported .ovpn file contains the authority and material from the moment of export, so a package whose certificate was later replaced will reject a client still carrying the old copy.

Transport is the other mismatch worth checking explicitly, because the profile and the package are edited in different places. Changing a profile from UDP to TCP without switching the package leaves the client waiting on a listener that does not exist, and the client reports the same generic handshake text it reports for everything else. Read the proto line in the file, then look at the protocol selected on the server page, and only then conclude that they agree. Reading the DSM firewall rules is the third check: a rule that allows the port from the local subnet but not from the internet is easy to leave behind after testing, and it produces a connection that works from the kitchen and fails from everywhere else.

Test the path a remote user really takes

A client inside the same LAN can reach the NAS without ever touching the forward, so it proves nothing about remote access. Use a phone with Wi-Fi disabled, or ask someone outside the network, and watch the router counter during that attempt. If the remote attempt fails while the local one succeeds, the forward or the firewall is still in the picture.

Record the working state while it works: external port, internal address, protocol, package interface, and the profile name and export date. When the handshake fails again, the difference between that note and the current configuration is the fault, and nobody has to rediscover which of the two ports the service uses.