Keep several vpnc tunnels up with per-profile probes

One vpnc profile reconnects cleanly after a roam; three of them take turns breaking each other's sockets, routes and resolver state. The packaged reconnect script is written for a single tunnel, so it restarts every profile whenever any one of them is down. Fixing that needs three things: separate local ports, a probe that tests a real service per profile, and one ordered chain that never runs two connects at once.

Give every profile its own local port

A vpnc profile left on the default local port competes with the profile already running. The second connection fails while the first holds the port, and the log line looks like a gateway problem rather than a local conflict. Set one line per profile file instead:

Local Port 0

Zero lets the kernel pick an ephemeral port, so two profiles can be established at the same time. Name each file for the gateway and group it serves, keep it readable by root only, and always connect with the matching profile rather than a shared default. A shared default file is how a script reconnects the wrong tunnel after somebody renames a profile.

Profile files carry credentials, so permissions belong to the design rather than to housekeeping. A world-readable file under /etc/vpnc hands the group secret to every local user. Keep the mode at 0600 with root as owner, and let the scripts read the file as root instead of copying secrets into arguments. That is also why a probe script should never echo the file contents into a log.

Probe one service per profile, not the tunnel address

An echo request to the tunnel's own address mostly proves the interface exists. It says nothing about whether the host behind it can still reach the application that justifies the tunnel. The interface can be up while the route beyond it has already gone.

Probe one service per profile with a short TCP timeout and reconnect only the profile whose probe failed. Two seconds separates a dead path from a slow one, and a TCP probe does not depend on ICMP being permitted anywhere along the way. Change only the target and the config name between profiles:

#!/bin/bash
set -eu
if ! nc -z -w 2 10.63.7.20 8443; then
  /usr/sbin/vpnc-connect ops.conf
fi

Save that as /usr/local/bin/ensure-vpn-ops.sh, make it executable, and repeat for each profile with only the probe target and the config name changed. A probe that answers a protocol question survives gateways that drop echo while still forwarding the services users care about.

Chain the probes so they never race

The packaged reconnect script can call your probes in sequence, and sequence is the whole point. Two vpnc-connect calls running together rewrite the routing table in an order nobody controls, which is how a healthy profile loses its route.

Keep the calls unbackgrounded, add a short settle delay for the underlying interface, and schedule the chain from root's cron table, since only root can rewrite the tunnel routes:

#!/bin/bash
sleep 20
/usr/local/bin/ensure-vpn-ops.sh
/usr/local/bin/ensure-vpn-lab.sh

Keep the order identical between the boot script and the cron entry, since a chain that runs in a different order at boot is effectively a different program. When one profile depends on another's routes, place the dependency first and let the later probe fail harmlessly while the earlier tunnel is still coming up.

Watch the log, then break one on purpose

Log one line per run with the profile, probe target, result and action taken. Without it, a tunnel that flaps every minute looks exactly like one that never dropped, and a probe bug is indistinguishable from a gateway outage. The log also proves the chain touched only the profile that failed. Write it somewhere that survives a reboot, or the evidence leaves with the process.

Then break a tunnel deliberately and watch what happens. Stop a single profile and confirm which processes and routes are still present:

pgrep -a vpnc
ip -4 route show | grep -E '10[.]63|10[.]90'
tail -n 5 /var/log/ensure-vpn.log

Only the stopped profile should be reconnected, and the other sessions should keep both their processes and their routes. If two profiles restart, your chain is running in parallel somewhere you did not expect.

What the watchdog must not do

A watchdog that restarts everything whenever anything fails looks healthy during testing and causes an outage in a maintenance window. Restart per profile, never globally.

Prefer a maintained client when the gateway offers one for the same protocol, and never hard-code a password into the script or pass one on a command line, because the profile file already holds the secret. If a task must run as an ordinary user once the tunnel is up, reduce privileges from inside the script rather than making the entire chain run under that account. Finally, re-read the routes after a reconnect, because a profile can return with a different assigned address and leave a stale route that no probe aimed at the tunnel address will ever notice.