Big Downloads vs the VPN: Choose the Path at the Request

"Send the big downloads around the VPN and keep the small replies on it" sounds like a routing policy, and it is not one. A router decides with information available at that moment: addresses, ports, protocol, firewall marks, an identified flow. The eventual size of a response is not in that set, because it has not arrived yet, and by the time the router could measure the bytes it is already carrying them. If you want steering by task, the decision has to move to a point that knows the task, and the checklist below is how you find out whether your design has such a point.

Why a router cannot size a response

At the moment a packet needs a route, the router has seen at most the first few packets of a flow. It can tell you who is talking to whom, over which protocol, with which mark, and how many bytes have passed so far. Bytes so far describe the past. The size of the reply is a property of a response that has not been generated, and for chunked or streaming responses it may never be knowable in advance.

Even the protocol's own metadata is weaker than it looks. HTTP allows responses whose length is not announced before the body, and a single user action can produce many requests whose combined cost is nothing like any one header value. Treating a header as the size of a task is a modelling error before it is a routing error.

Check 1: is there a decision point that can choose a path?

Name the component that will make the choice, then name what it knows at the moment it chooses. An updater that fetches from a documented set of hosts knows its own purpose before the transfer starts. A download service that consults metadata and then opens the request knows the task before the connection exists. A cooperating proxy can apply request-level policy or cache the response without pretending to be a router.

If the answer is "the gateway", the checklist has already failed. A gateway can classify known flows by destination, mark or client segment and send them down a defined path from the first packet, which is often good enough for an approved updater and never enough for an unknown response size.

Check 2: what does the metadata actually tell you?

Application-aware steering is only as good as the fields it reads. The response-length rules in HTTP semantics permit cases where the length is unavailable at the start, and content encodings make the transferred byte count different from the representation size anyway. A policy that switches paths when a header crosses a threshold will therefore behave inconsistently, and mostly on exactly the large transfers it was meant to catch.

What application metadata can support is a decision about a task: this workflow is a bulk sync, this one is interactive, this one is a software update from a known origin. That is a policy written in terms of intent, and it is stable enough to test. "This response will be bigger than forty megabytes" is not that kind of policy.

Check 3: does the flow already exist?

Routing can change for a flow without changing the connection's identity, which is a distinction worth keeping straight. A TCP connection is identified by its endpoint addresses and ports, and the packets can take a different path if the routing table changes underneath them. What cannot change invisibly is the network address translation state: moving traffic to a different exit usually changes the public source tuple, and the far end will see a different peer with no matching state.

conntrack -L -p tcp --dport 443 | head -5
ip route get 203.0.113.10
ss -tn state established dst 203.0.113.10

So the practical rule is that a mid-connection move from one egress to another is a new relationship, not a migration. If the design requires the same connection to continue, the decision had to be made before the first SYN, and no packet rule added later can reconstruct it cleanly.

Check 4: is the real requirement latency, not egress?

A great many requests for "send bulk traffic elsewhere" are actually complaints that a large transfer makes interactive work feel slow. That is a scheduling problem, and it has a scheduling answer: fair queuing or a supported bandwidth policy on the existing path keeps a bulk flow from starving a chat message without moving either of them.

tc qdisc replace dev eth0 root cake bandwidth 100mbit

Addressing the symptom this way keeps the egress decision where the security policy put it, and it is reversible in one command. Changing the exit for a whole class of traffic is a policy change with authentication, logging and access-rule consequences, so it should not be the first tool reached for a latency complaint.

If the requirement genuinely is a different internet exit, say so in those words and implement it at a supported boundary: a new connection from an approved workflow, or an application that opens its own request through a chosen egress. Then record which tasks use which exit, so the next person does not have to infer it from a routing table.

Keep the checklist in the change record: the decision point, the field or task that drives it, whether the flow already exists, and whether the measured symptom was latency or egress. Four questions, and a policy that answers all four is one you can operate rather than defend.