VPN Relay Stops Accepting SSH: File Descriptor Limits

A WireGuard handshake still completes, the peer still shows a recent endpoint, and yet a new SSH session through the relay behind it hangs with no banner. That combination points away from the tunnel and towards the process that accepts the socket. A relay such as socat sits in the application path with its own descriptor budget, its own children and its own inherited limits. The walk below starts at the client and stops at the first component that cannot open another socket.

Follow the Path Before Changing a Number

Trace one failing request instead of reasoning about the tunnel as a whole. The client writes into the tunnel, the VPN daemon decrypts and routes, and the relay accepts on the far side. Each step consumes descriptors, and the one that usually runs out is the last.

Start with a request you can reproduce and time it. If the TCP handshake completes but no application banner arrives, the relay took the socket and then stalled in a child or on its upstream leg. If the handshake itself never completes, the cause is routing, firewall or listener state rather than a descriptor ceiling. Record which of the two shapes you observed, because they point at different processes.

Count the hops that duplicate work as well. A forking listener hands every accepted socket to a child that keeps its own copy, so the real total is a property of the whole tree, not of the parent alone.

Read the Limit the Relay Inherited

The only limit that matters belongs to the running process, and it was fixed by whatever launched that process. Typing ulimit -n in a shell describes the shell. Ask the service manager what it applied, then confirm on the PID.

systemctl show -p MainPID -p LimitNOFILE vpn-relay.service
cat /proc/$(systemctl show -p MainPID --value vpn-relay.service)/limits

The limits file prints soft and hard ceilings as the process sees them. A low soft value under a generous hard value means the daemon is allowed to raise its own ceiling, and some of them do that at startup; equal low values mean the unit file is the source.

Drop-ins load in lexical order and later files win, so systemctl cat vpn-relay.service is the authoritative view of what applies. Editing a file the manager never reads is the most common reason a raised number has no effect.

Separate Steady Concurrency From Retained State

Two processes can hit the same ceiling for unrelated reasons, and the remedies differ. Demand that tracks useful concurrent sessions is a capacity question. Growth that continues after the work ends is a leak or a set of sessions nobody closed.

Measure at idle, at your normal peak, and again a few minutes after load stops. Counting open entries with ls /proc/PID/fd | wc -l against the soft limit tells you the gap, and a count one or two below the ceiling is a hard stop rather than a slow path.

When the count refuses to fall, look at socket states rather than totals. Descriptors sitting in CLOSE_WAIT mean the peer closed and your side never did, which is an application lifecycle defect that a larger limit only postpones. Bound the child model too: a forking relay with no cap on concurrent children exhausts descriptors, process slots and memory together.

Check the New Value on the Live Process

After editing the unit, reload the manager and restart the service, since a reload alone never replaces limits already inherited by a running process. Verify on the process that is actually serving traffic.

systemctl daemon-reload
systemctl restart vpn-relay.service
awk '/Max open files/ {print}' /proc/$(systemctl show -p MainPID --value vpn-relay.service)/limits
ss -tnp state established '( sport = :2222 )' | wc -l

The awk line returns the new pair; if it still shows the old one, the drop-in you edited is not the one being loaded. The ss count is your workload check, so open the number of sessions you expect and watch it climb. A count that keeps rising after those sessions close means the limit was never the whole problem.

One caveat belongs with any large increase. Applications that poll with select() cannot watch descriptors above FD_SETSIZE, which is 1024 on typical builds, so a ceiling raised past that can produce failures that look unrelated. The systemd execution reference carries the same warning for services it starts.

Keep the Restart Away From Tunnel Availability

Restarting the relay drops its active sessions, and that is a separate event from the VPN going down. Order the change so the tunnel stays up while the relay cycles, and warn anyone whose long-lived session crosses it.

Rollback is the previous unit file plus another restart, and that file stays in place until the new value has survived a working day. Then close the loop with the request that originally failed: one new SSH session through the relay, one transfer long enough to matter, and a descriptor count that returns to its idle level.