One Lab Server, Three Trust Roles, One Failure Domain

Everything works until the build runner needs the same host socket the VPN guest uses, and then nobody can explain why a job can read the gateway's keys. Everything was installed on one machine because it fit, and every guest was given broad access because that was the quickest way to make it work. The result is a host whose trust boundary is the power cable. Splitting the roles takes less work than it sounds, and it starts with naming them.

Three Roles That Currently Share Everything

A VPN entry, a set of private services and a build runner want different things. The first needs a public port and a narrow private-side network. The second needs persistent storage and application secrets. The third executes code that somebody else wrote.

Giving all three the same access is what makes the symptom confusing, because any of them can then affect the others. A build job that can reach the hypervisor socket is a job that can stop the VPN. A services guest with a route to the management network can change switch configuration. The privileges are the problem, and the process list is only where it shows up. Naming the roles first gives each privilege somewhere to live.

The sketch below is a starting point for that split rather than a promise of high availability. Treat it as a hypothesis to verify, not as the finished design.

Split Three Roles, Not Three Processes

Three guests on one host give three privilege sets and three network positions. They do not give three failure domains, and the design should say so out loud.

The separation only helps if a compromise in one guest does not automatically grant the others, which is a hypervisor setting rather than a naming convention. Calling a VM gateway does not stop another guest from reaching the host. Read the virtual switch configuration instead of the hostname, then confirm that each guest sits where you think it does.

host
├─ vpn-gw      10.60.10.2/24   public 51820/udp only
├─ services    10.60.0.10/24   no route to the management network
└─ ci-runner   10.60.20.2/24   outbound only, no host socket

Write the topology down and then verify it with the tools that own it. A guest that appears on the wrong bridge is a finding rather than a detail. Correct the network before changing any service configuration.

Let the VPN Guest Open Only the Doors It Owns

Publish the tunnel port at the router and nothing else. Guest administration ports belong on a management path, not on the internet, and the hypervisor's own interface belongs there too.

The private-side network should reach the services that remote users need and stop there. Application authentication stays switched on, because a tunnel is transport and not a login. Keep the VPN guest's routing table small enough to describe in one sentence. If you cannot describe it that simply, the boundary has already blurred.

virsh net-list --all
virsh net-dumpxml lab-private
sudo nft list ruleset | grep -n -E '51820|10.60'

Anything listening on the host that is not needed for management should not be listening at all. That includes the hypervisor API, which is easy to leave exposed on a public interface.

Build Jobs Need a Different Kind of Boundary

A CI job runs repository-controlled commands, so its trust depends on who can push code. A shell executor runs those commands as the runner's user on the host, which is the widest boundary available.

Container executors are narrower, and a privileged container is not one of the narrow ones. Keep the runner in its own guest with its own network scope, and give it only the artifact and deployment access the jobs need. Disposable workers reduce retained state; they do not reduce reachability or secret exposure. Both of those still need explicit limits.

The services guest owns the data, so its backup plan is the one that has to work. Keep repositories, uploads and the database in the same schedule. Test a restore instead of reading the job status.

Plan for the Failure All Three Share

One host is one failure domain. A reboot removes the tunnel and every private service at the same moment, which is exactly when a way in matters most.

Keep console access that does not depend on the VPN guest, whether that is a physical console, an out-of-band management interface, or a provider console. A snapshot stored on the same disk is not an independent copy, and neither is a backup that runs inside the guest it protects. Then test cold start as a sequence rather than a checklist: host, virtual networks, tunnel path, and one named application under an ordinary account.

virsh list --all
systemctl is-active libvirtd
curl -sS -m 5 -o /dev/null -w '%{http_code}\n' https://10.60.0.10/
ss -ltnp | grep -E ':51820|:443'

Ordering process starts proves nothing about routes and DNS, so test the path rather than the unit states. The path is what a remote user actually needs.