The Routing Rule Reads the Packet Mark, Not conntrack

Four checks separate a working mark rule from a working route: which mark the rule reads, whether the packet still carries it at the lookup, whether a connection mark was restored onto later packets, and whether the matching rule's mask and priority allow it to fire. A counter that increments proves only that some rule matched something. It says nothing about what the packet carried when the kernel chose its path.

The two marks live in different places and expire on different schedules.

Two Marks, Two Lifetimes

A packet mark belongs to one skb, travels with it through the hooks and is consumed by policy routing's fwmark selector. A connection mark belongs to the conntrack entry, persists across the packets of that flow, and is invisible to the routing rule until something copies it back. Saving a packet mark into the connection is a store; restoring it is a separate action. Neither happens implicitly.

That is why a ruleset can look consistent while the routing decision sees zero: the value was saved, and nothing restored it.

Copying Between Them Is a Step You Write

The two statements that matter run in opposite directions: one stores the packet's mark into the connection, the other loads the connection's mark onto the packet. Which one you need depends on when classification happens relative to the flow. A design may need only the packet mark, and adding conntrack marking because a workaround used both is how a simple policy acquires a second failure mode.

ct mark set meta mark
meta mark set ct mark

Write the direction down next to the rule, because it is not obvious from the syntax at a glance. The first line stores, the second restores, and neither implies the other.

Classify Before the Lookup That Consumes It

Locally generated traffic and traffic forwarded from a LAN traverse different hooks and reach routing decisions at different moments. An output rule that works for the router's own connections does not prove that a forwarded client's packet was marked before its lookup. Put classification where the lookup that consumes it has not yet happened, and check whether later rules in the same chain overwrite the value. Ordering within a chain is part of the configuration, not an implementation detail.

Connection-state persistence needs an explicit plan too. If only the first packet of a flow is classified, later packets need the saved mark restored, or they will take the unmarked path. That is the difference between a mark that describes a connection and one that describes a packet.

Ask the Route a Marked Question

ip route get 10.77.0.10 mark 0x2

Inspect the rules and the table, then query the route with the mark included, plus the source or incoming interface when those affect the policy. A plain unmarked lookup answers a different question and will happily report the path you did not want. Check the rule's mask and priority as well as its value, and confirm the selected table contains a usable route.

An incrementing counter is evidence that some rule matched some packet. It is not evidence that the final packet carried the intended mark at the right decision point.

Read the mark, the direction, the timing and the mask. Four checks, and the route stops being a mystery.