nftables cgroup Bypass Also Needs a Source Address Fix

Marking traffic from one cgroup is the easy part. Three separate decisions have to line up before a single application leaves through the physical interface while everything else stays in the tunnel: which socket belongs to the exception, which routing lookup the mark triggers, and which source address the packet carries when it leaves. Most failed attempts get the first right, half-get the second, and never look at the third.

Three decisions, not one

Classification answers who is exempt. Routing answers where the marked packet goes instead. Source selection answers what the far end will see and where the reply should return.

These fail independently. A packet can be classified, rerouted onto the physical interface, and still carry a private address that no upstream router can return, which produces a timeout that looks exactly like a firewall block. Nothing in the classification step warns you about that. Nothing in the routing step does either.

The order matters too. Classification happens when the packet is generated, routing happens after that, and translation happens at the point of egress.

If you cannot name which of the three stages is broken, you are not debugging, you are rearranging rules and hoping.

Match a cgroup that will still exist tomorrow

nftables can test a socket's cgroup v2 membership, which is genuinely useful because it follows the process rather than the destination. The catch is the identifier. A desktop application launched from a menu often lives in a transient scope whose name changes on the next launch.

nft list ruleset
systemctl status session.slice 2>/dev/null
cat /proc/self/cgroup

Copying a scope name out of one session and hardcoding it produces a rule that works today and silently stops matching tomorrow. Put the application in a deliberate, stable execution context instead, such as a dedicated scope or unit you control, and match that. The rule then survives reboots and package updates. It also becomes something a colleague can reason about without your session history.

Also be precise about which mechanism you are using. Cgroup v1 net_cls classification and cgroup v2 socket matching are different things, and a similarly named field in documentation does not make them interchangeable.

A route chain is what triggers the second lookup

For locally generated traffic, the chain type decides whether anything happens after the mark is set. A filter-type output chain can mark a packet and the kernel will keep using the route it already chose. A route-type chain is the mechanism that asks for a new lookup.

table inet bypass {
  chain mark {
    type route hook output priority mangle; policy accept;
    socket cgroupv2 level 2 "app.slice" meta mark set 0x1 comment "app bypass"
  }
}
ip rule add fwmark 0x1 lookup 200
ip route add default via 203.0.113.1 dev wlan0 table 200

The mark then needs a policy rule that sends it to a table, and that table needs to be complete: a default route, any local destinations the application still requires, and an upstream gateway that actually answers.

Choose the mark value deliberately. WireGuard uses its own fwmark for outer transport traffic, and landing on the same number by accident can interfere with the tunnel's own routing exception in ways that are painful to diagnose.

The source address does not follow the route

A socket that already selected a source address for the tunnel path keeps that address after the reroute. Nothing in the routing change rewrites it, which is why the packet leaves the physical interface with a private source the upstream network cannot return.

The visible symptom is an application that appears to send data and never receives an answer, and a firewall log that shows nothing because the packet never came back.

The narrow fix is a source translation rule scoped to that application's mark and its real output interface, integrated with whatever NAT already exists rather than replacing it:

table inet bypass {
  chain postrouting {
    type nat hook postrouting priority srcnat; policy accept;
    meta mark 0x1 oifname "wlan0" masquerade
  }
}

Translate the exception and nothing else. A blanket masquerade rule fixes the symptom and destroys your ability to tell which traffic took which path. Match on the mark, the real output interface and, if you can, the application's own address. Broad rules are easy to write and almost impossible to audit later.

What still escapes the exception

Name resolution may come from a separate resolver process in a different cgroup, so classifying the browser's socket does not classify its lookups. If the resolver's queries stay in the tunnel while the connection leaves outside it, split-horizon answers can mismatch the path. That mismatch produces failures that look like DNS bugs and are not.

Child processes inherit the cgroup in most setups, but a helper that moves itself into its own scope does not. Check rather than assume, especially for anything that re-executes itself.

Finally, decide what should happen when the tunnel is down. An exception that silently becomes a full bypass is a different security posture from one that fails closed, and only one of them is what you meant.