Failover Test: Does an Established TCP Stream Survive?

Two outcomes hide inside the sentence the backup works. A new request succeeds, which is recovery. A transfer already in progress continues, which is continuity. Those are different promises, and most failover designs deliver the first while the second quietly times out. Decide which one you need before choosing a mechanism, and write the test so it can tell them apart.

Two results that look like one success

Imagine a client at 10.90.0.42 with a connection to 198.51.100.20:443. The primary exit translates that to one public address and port, and the backup exit translates it to another. The client's inner address never changes, which is why people expect the connection to survive.

The server does not see an inner address. It sees the translated endpoint, and that endpoint is different after the switch.

So the old stream times out while a fresh connection to the same service works immediately. Both observations are true at once.

What the server sees when the exit changes

A TCP connection is identified by its four-tuple, and changing the source address or port creates a different connection as far as the receiving host is concerned. Packets that arrive from a new source on a connection the server does not know are dropped or answered with a reset. Nothing in the VPN client's address preserves the association across that change.

Stateful failover exists precisely to avoid this, and it works by exchanging translation and connection state between the forwarding devices. That is a feature of the pair, not of the tunnel, and it has to be designed and tested rather than assumed.

Check whether the pair actually keeps state before you promise continuity to anyone. The commands below show the current state on both sides.

conntrack -L -d 198.51.100.20 2>/dev/null | head
ss -tin dst 198.51.100.20
tcpdump -ni any host 198.51.100.20 -c 10

Why the same public address is not enough

Keeping one public address across both paths is a good start, because the server then sees the same source endpoint. It is not sufficient on its own. The forwarding device still needs the translation state, the firewall state and a usable return path, and those have to exist on whichever device is carrying the traffic after the switch.

RFC 3022 describes address translation and treats state exchange between translating devices as part of a backup design. A floating address or a copied profile moves the configuration without moving the state.

That is why a test that only pings through the backup proves so little. Ping has no connection state to lose.

Decide how strong the requirement is

If users can reconnect after a failure, the design is a recovery design and the acceptance test is a fresh request. If a file transfer must survive, the design needs state synchronisation and a much more careful test. If a remote session must survive, the requirement is stronger again and may need an application-level mechanism rather than a network one.

Write the requirement as a sentence with a measurable outcome, such as the upload completes without restarting or the operator reconnects within a minute. Then pick the cheapest mechanism that satisfies it, which is frequently the recovery design.

Say explicitly what happens when the primary comes back, because an automatic failback can interrupt a connection that had just recovered on the backup. That case belongs in the test plan, not in a footnote.

Check the old stream and the new request

Start a harmless long-lived transfer through the primary, then a separate fresh request to the same service. Record the application context, the observed exit address and the time. Break the primary in an approved way and keep watching the original transfer while issuing the new request through the backup.

Report three values: whether the old stream continued, reset or timed out, when the new operation first worked, and whether failback interrupted anything. Those three lines are the acceptance record.

ss -tin dst 198.51.100.20
curl -sS -o /dev/null -w '%{http_code} %{time_total}\n' https://198.51.100.20/
tcpdump -ni any port 443 -c 20

If the workload writes data, do not repeat an uncertain operation blindly after a timeout. The write may have landed before the answer went missing. Ask the application for the result, or retry only where the operation is idempotent. Then write the acceptance statement in three parts: new operations recover under the exit policy, existing streams have the documented interruption behaviour, and the application resumes safely. That statement is testable, and it is honest about what the design does not promise.