A kill switch may or may not cover the window before its app starts, and the difference is decided by operating-system policy rather than by the client's marketing page. An automatic-launch checkbox proves that something is scheduled to start. It does not prove that blocking rules exist at the moment the network comes up. Those are two events, and only one of them protects traffic.

Work the checklist in order, because each step narrows what you are actually testing.

Where the block actually comes from

There are two families of implementation. Either the platform installs the block, because it knows about the tunnel before any app runs, or the client installs the block itself after it starts. The first can exist before login. The second cannot exist before its process does, however early that process is scheduled. Identify which family you are relying on before testing anything.

On Android, always-on VPN combined with the block-connections-without-VPN setting is platform policy, configured in system settings rather than only inside a client. Managed devices can have it set for you. A client that merely starts at boot is not the same mechanism.

Separate the window from the service

A closed window is not proof that enforcement has stopped, and an open window is not proof that a tunnel is ready. Many clients separate the interface, the daemon and the tray application. The tray is what you see; the daemon is what holds the rules.

Test the component, not the icon.

Test the startup interval safely

The interval to test starts at reboot or login and ends when the tunnel authenticates. Background applications connect during that period, and update checks are the usual offenders. Use harmless activity and the vendor's documented procedure rather than blocking your own access. Then repeat after a normal reconnect, because a reconnect path and a cold-start path run different code.

A test performed after the tunnel is already up proves nothing about startup.

Read the reboot behaviour if persistence is the requirement. A setting can survive in configuration while its enforcement begins later, and both facts matter: the saved policy, and the moment it becomes active.

Read the policy per platform

Android exposes always-on and blocking as separate switches, and both have to be on for pre-launch protection. Windows has an always-on device tunnel concept configured through policy, which is not the same as a user-mode client starting early. OpenVPN Connect documents its own kill-switch behaviour per platform, and the documented scope is what you test. Do not carry a setting name from one system to another and assume the semantics travelled with it.

A profile requiring an interactive password has different startup dependencies from a machine tunnel.

What to record

Write down which mechanism is in force, at which stage it becomes active, and what traffic was attempted before that stage. Record the platform version, because this behaviour changes between releases and between managed and personal configurations. Note whether the client or the system owns the policy, since that decides who can fix a gap. Then re-test after any client or OS update.

If the gap matters, the durable fix is platform policy, not a shorter launch delay.

That is the answer to the original question, stated as a condition: a kill switch protects traffic before the app starts only when the block is installed by something that exists before the app does.

Nuŋlɔɖi siwo ku ɖe eŋu