VPN service fails at EXEC: compare unit, binary and sudoers

Yesterday the tunnel worked, and this morning the unit is dead with Failed at step EXEC spawning /usr/local/bin/vpnclient in the journal. Credentials, routes and the account never enter that sentence. systemd resolved ExecStart to a path, asked the kernel to execute it, and the kernel refused, most often because nothing exists at that path or it is not executable. Treat the fault as a file-and-argv problem until the parsed command proves otherwise.

Why EXEC fails before authentication starts

systemd does not run ExecStart through a shell. It forks, builds an argument vector from the line, and calls execve on the first element. If that call fails, the service never reaches the VPN client's own code, so no amount of profile editing or credential checking can help.

The status code separates this family from ordinary crashes. A unit that exits with status=203/EXEC never started a process; a unit that starts and then exits with status=1 may have failed authentication, DNS or routing. Read the code before reading the client log, because the client log will be empty in the first case.

The path in the journal is the one systemd used, not necessarily the one you edited. Drop-ins and generator output can override the vendor file, and a stale daemon-reload leaves the old line in memory. That is why the first diagnostic is to ask systemd what it thinks the command is, rather than to open the file you remember changing.

Three places the binary path is written down

A VPN client installed from a distribution package, a vendor script and a release tarball can land in three different prefixes. The unit, the privilege rule and the installer each recorded one of those prefixes, and the failure appears when two of the three disagree.

A concrete mismatch looks like this: the vendor script installs to /usr/local/bin, the tarball unpacked under /opt, and the unit still names the old /usr/bin path from a previous package. Only one of those files exists, and it is rarely the one in ExecStart.

Decide which path is canonical before changing either file. If the distribution package owns the binary, the package path is canonical and the unit should follow it. If the vendor script owns it, the same rule applies to the vendor prefix. Copying a binary between prefixes creates two copies that drift on the next upgrade.

When a quoted command becomes a file name

A second failure mode hides behind the same journal line. Someone writes this into the unit and expects the client to connect during boot:

ExecStart=/usr/bin/"vpnclient connect office"

systemd splits the value into words, so the first word becomes /usr/bin/vpnclient connect office with the quotes removed as grouping syntax. The kernel then looks for a file whose name contains spaces, which does not exist. The error is identical to a missing binary even though the binary is installed.

The supported form is a real executable followed by separate arguments, for example ExecStart=/usr/bin/vpnclient connect office. If the workflow genuinely needs a pipeline or a chain, put it in a script and call the script, because ExecStart has no &&, no pipes and no glob expansion.

Fix the unit, not the filesystem

The tempting repair is a symlink from the expected path to the real binary. It works until the package updates and the link points at a removed file, or until an SELinux label no longer matches the target. Prefer editing the unit through a drop-in so the vendor file stays untouched and upgrades remain predictable.

The privilege rule has to match the same path and the same subcommands. A sudoers entry for /usr/local/bin/vpnclient does not authorize /usr/bin/vpnclient, and an entry that allows every argument lets any local process reconfigure or stop the tunnel. Grant the two verbs the unit needs and keep teardown out of it if stopping the service already closes the interface.

This is where the comparison pays off: the unit, the package database and sudoers should all name one path. When they do, the EXEC error disappears without a reboot and without a copy of the binary in a third location.

Prove the parsed command before reboot

sudo systemctl daemon-reload
sudo systemctl cat vpnclient.service
systemctl show -p ExecStart vpnclient.service
systemd-analyze verify /etc/systemd/system/vpnclient.service
sudo systemctl restart vpnclient.service
systemctl status vpnclient.service --no-pager

systemctl show is the decisive output because it prints the argv systemd actually parsed, including any drop-in. systemd-analyze verify catches a missing or non-executable path before you restart, and the status command after restart tells you whether the process now exists.

If the unit starts and then fails, you have left the EXEC family and entered the client's own error handling. Check the journal for the client's message, not for another execve failure. Keep the unit diff and the sudoers diff together in the change record so the next upgrade does not silently reintroduce the split path.