New connections on a new VPN exit, old flows left alone

A planned drain moves only unclassified flows. Compare that with a route swap, and see the marks, tables and conntrack state that make it hold.

Two healthy VPN exits, one long upload already running, and a requirement that the next connection uses the other tunnel. There are two ways to satisfy that, and only one of them is safe on a live gateway. Swapping the default route moves every packet, including the ones that belong to the upload. A planned drain changes the choice for new connections while leaving the tracked flows on the path they started on.

Two ways to switch an exit

The route swap is one command and it is reversible in seconds, which is why it gets used during incidents. It is also indiscriminate: packets from the existing upload suddenly leave through a different tunnel, with a different public source address and separate translation state.

If the far end tracks flows by source address, the upload may simply stop being recognised. The drain avoids that by making the decision once per connection and storing it, so the old upload keeps its exit until it finishes. That is the whole value of the exercise: the far end sees no change it has to interpret. It keeps receiving packets that match the connection it already knows.

The cost is complexity: two routing tables, a classifier that runs at the right hook, and marks that must be coordinated with whatever else on the box uses them.

Give each exit its own table

Reserve a routing table per exit and distinct mark values or bits for the two choices. Each table needs the routes appropriate to its role, not just a name, and it needs a deliberate failure outcome for the missing case. An empty table that silently falls through to the main table is how traffic leaves through an exit nobody selected.

ip rule add fwmark 0x10/0x10 table 101
ip rule add fwmark 0x20/0x20 table 102
ip route add default via 192.0.2.1 dev wg-a table 101
ip route add default via 198.51.100.1 dev wg-b table 102

Add a blackhole route at the end of each table so a lookup that finds no match fails loudly instead of escaping to a default you did not choose.

Choose once, at the first packet

Classify after connection tracking exists and before the route lookup that matters. For a new, unclassified LAN connection, pick the exit and save it into connection state. On later packets, recover the saved value into the packet mark that policy routing reads.

The subtle part is what new means. Conntrack can still report NEW for early packets and retransmissions before a reply is seen, so a classifier that keys only on that state can overwrite a choice it already made. Check that no exit is stored for the connection before assigning one.

nftables keeps packet and connection marks separate, and the metainformation guide documents the explicit copy between them. A connection mark does not become the packet mark your rule matches by itself, which is the single most common reason a carefully written classifier has no effect.

Scope restoration to the direction and traffic you mean. Applying an internet-exit mark to returning packets can send replies toward the exit instead of the LAN, and the symptom is a one-way connection that looks like a firewall problem. Read the reply path before you touch a firewall rule, because the packet is being routed, just not back to the sender.

Four phases of a drain

  1. Prepare B without touching A: routing, forwarding, authorization, NAT and the return path. Install its table and selection policy through the gateway's existing configuration owner.

  2. Switch the classifier so new, unclassified connections choose B, preserving saved choices. An atomic nftables transaction avoids a window where old state has to be cleared.

  3. Leave A up while its flows finish. Its interface, table, remote exit and NAT mappings are still part of those flows, and editing a NAT rule does not rebuild bindings that already exist.

  4. Retire A when the drain condition is met, judged from connection state and application behaviour rather than one quiet interval.

What a drain cannot promise

A browser request is not a new transport connection; pooling keeps sending requests over the old one. Conversely, a UDP flow can age out of conntrack and a later packet gets classified fresh. The drain follows tracked flows, not an application's idea of a session.

If A fails before its flows finish, the stored mark does not save them. They reconnect through B, or they do not reconnect at all. A planned drain buys continuity while the original path is healthy; it is not state replication between independent exits.

Verify both halves

Start a long-lived permitted connection before the change and a genuinely fresh one after. Confirm which tunnel each uses and which public source address the far end sees, then check that return traffic comes back.

Do not flush the conntrack table to make the new choice appear. That destroys exactly the state the drain is supposed to preserve, and it turns a controlled migration into an outage with a clean-looking test.

Winndannde jowitiiɗe heen