PAM auth works by hand but fails as an OpenVPN service

Authentication through PAM can succeed when you run the client by hand and fail with AUTH_FAILED once systemd starts the same profile. The password does not change between those runs; what changes is the set of constraints around the process that asks PAM the question. PAM answers through a helper whose ability to change its own credentials is exactly what a hardened unit removes. Find that helper and the boundary it needs before you touch a capability list.

unix_chkpwd is the binary that answers the PAM question

With local password authentication, pam_unix rarely reads the hash file itself. It executes unix_chkpwd, which distributions install setuid root or setgid shadow so that one password can be checked on behalf of a process that has no business reading /etc/shadow. Remove the ability to gain that identity and the helper cannot complete the check, so the module returns a plain authentication failure.

The failure then looks like a credential problem in every log the client can see. That is the trap: the client reports what PAM told it, and PAM reports only that the stack said no. Running the helper by hand as root also proves nothing, because it exercises a privilege path the service never uses. Read the unit that systemd actually loaded, then read the system log for the attempt it made. Those two views separate a sandbox problem from a wrong-password problem.

Read the unit before you read the password log

A packaged unit can carry a drop-in you did not write, and the effective configuration is the merged result. Print the unit as systemd sees it, along with the sandbox settings that decide whether the helper can elevate itself. Read-only inspection is enough at this stage.

systemctl cat [email protected]
systemctl show [email protected] -p User -p Group -p CapabilityBoundingSet -p AmbientCapabilities -p NoNewPrivileges
journalctl -u [email protected] -b

The first command shows where the unit and its drop-ins come from, which matters because an override added months ago outlives the memory of why it exists. The second prints the sandbox values that the next two sections depend on.

Compare what you see with a working host of the same distribution and version. A difference in User, CapabilityBoundingSet or NoNewPrivileges is a concrete lead; a difference in prose comments is not. Write the values down before changing anything, because a later comparison against a moving target proves nothing. If the service runs as an unprivileged account with an empty bounding set, the helper has almost nothing to work with.

Credential transformation, file access and policy are three boundaries

Start with what happens during exec. A setuid binary or a file carrying file capabilities triggers a credential transformation, so the resulting identity is not a continuation of the caller. The bounding set caps what such a binary may acquire, and NoNewPrivileges blocks setuid, setgid and file-capability gains outright. Any one of those three can stop the helper while leaving every other service function intact.

Next, separate broad discretionary bypass from narrow read access. CAP_DAC_OVERRIDE skips discretionary checks almost everywhere, while CAP_DAC_READ_SEARCH covers reading and directory traversal with its own additional powers. They are different tools with different blast radii, and neither is a spelling of let PAM read the shadow file.

Finally, ask whether the mandatory policy permits the action at all. A Linux capability does not override SELinux, and holding one proves nothing about a type enforcement rule. Treat a file-access refusal and a capability refusal as different evidence, because they name different fixes and often different owners. Read every field of the record rather than the single word denied. Which executable took the action, which domain it ran in, what type of target it touched, what class of object was involved and which permission the policy refused each narrow the answer a different way.

Pull the denial out of the audit log

On an enforcing host, the audit trail is the only place that records a refusal. Query the recent events and read them against the timestamp of the failed login, not against the time you started investigating.

ausearch -m AVC,USER_AVC -ts recent

An empty result is not evidence of innocence. It means either that auditing is not collecting, or that your window missed the attempt, and both possibilities need to be excluded before you rule the policy out.

Red Hat SELinux troubleshooting guidance puts label and configuration checks ahead of generating allow rules, which is the right order for this problem too. A mislabelled helper binary and a policy gap produce similar denials but need different repairs. Generating an allow rule for a labelling mistake hides the fault and leaves the next package update to reintroduce it. Confirm the label on the installed helper before writing any policy.

Prove the change with one accepted and one rejected login

Change only the boundary you demonstrated, then test both outcomes. An account that should authenticate must succeed, and an account with a wrong password must still fail. A change that makes every attempt succeed has removed authentication rather than repaired it.

Run a real connection attempt and read the journal immediately afterwards. The line that proves the fix is the pam_unix(openvpn:auth) entry for that attempt: it should read authentication success, and no AVC record should carry the same second.

journalctl -u [email protected] -n 30 --no-pager
ss -tnp | grep openvpn

A working tunnel also needs a reconnect, because sandbox changes break the second connection more often than the first. Check the established socket after the reconnect and confirm the same unit is still serving it.