Rotate a VPN watchdog log without breaking the writer

A VPN health check that appends one line a minute quietly creates a second duty: deciding how those lines leave the disk. Fixing the tunnel does nothing for that duty, and renaming the file does not redirect a process that still holds the old descriptor. Answer the sink question first, then make rotation match the way the writer opens it.

Choose the sink before the rotation policy

A systemd unit that already sends its output to the journal keeps retention in one place. journald enforces limits such as SystemMaxUse and MaxRetentionSec, so a watchdog that logs there inherits a bound without any extra work. Adding a second unbounded file next to it doubles the maintenance for no gain.

A dedicated file earns its place when something else reads it: a parser, an audit export, or a page that expects a stable path. Once that decision is made, the file needs an owner, a mode, and a rotation policy written down together, because the three are only correct as a set. A path that anyone can create is also a path anyone can replace, so the directory permissions belong in the same note. Write all of it next to the check and the next edit starts from the same facts.

Write the stanza to match the writer

Start from a writer that opens the path on every invocation, appends a line, and exits. That pattern survives rotation without any cooperation from the process, because the next run opens whatever file currently sits at that path. The stanza below keeps a week of versions and recreates the file with the owner and mode the writer needs:

/var/log/vpn-watchdog.log {
    daily
    rotate 7
    create 0600 root root
    missingok
    notifempty
}

The logrotate manual defines the count as the number of rotated versions kept, not as a guaranteed seven days of history, and the daily keyword is evaluated when the job runs rather than acting as a scheduler of its own. On most distributions a systemd timer or a cron entry already owns that invocation, so check which one exists before adding a second. Matching ownership matters more than the retention count. Copying a root-only mode onto a file written by a service account stops logging at the next rotation, and the failure appears hours later as a stale timestamp rather than as an error. Confirm the writer's user before you copy a stanza from somewhere else.

A writer that never reopens needs a different fix

A long-running process holding an open descriptor keeps writing to the renamed inode. Space is not freed and the visible file stops growing, which looks like a broken rotation until you check the inode. Either teach the process to reopen, or make rotation copy and truncate instead of rename.

The reopen path is the cleaner one where the software supports it: send the documented signal or run the documented command, and let the process open the new file itself. The alternative is copytruncate, which copies the contents to the rotated name and then truncates the original in place, so the descriptor stays valid but writes that land between the copy and the truncate are lost.

For units started by systemd, point StandardOutput at the journal or at a file and let the unit own the lifecycle. A shell loop that appends through tee -a with no size bound is the arrangement that fills a partition during an outage, because a failing check often logs far more than a healthy one.

Test the rotation without waiting for the timer

Debug mode reports what would happen and touches nothing, which makes it the safe first run after an edit. Force a real rotation once you are satisfied, then confirm that the next log line lands in the new file rather than the renamed one:

logrotate -d /etc/logrotate.d/vpn-watchdog
logrotate -f /etc/logrotate.d/vpn-watchdog
ls -l /var/log/vpn-watchdog.log*
tail -n 3 /var/log/vpn-watchdog.log
df -h /var/log

Read the output in that order. The debug pass should name the file and the actions it intends; a forced run should produce the rotated name you expected; the tail should show a fresh entry after the writer runs again. A growing df value with a shrinking visible file means the descriptor problem described above.

Repeat the forced run twice to see how retention behaves when versions already exist. Deleting the oldest file is the expected result, and a missing deletion usually means the count or the wildcard in the rotated name does not match what logrotate created.

Keep a rollback and a written record

Save the previous stanza before editing, and note which scheduler runs it, because a rollback that restores the file but not the timer reproduces half the original problem. Rotate once by hand after restoring, then confirm the log path is writable by the writer's account. If the service runs under systemd, reload the unit after any change to its output redirection, since the running process keeps writing to the old target. Read the next few log lines yourself instead of trusting that the unit came back cleanly.

Record the mode, the owner, the retention count and the expected disk budget next to the check itself. When the check later changes from a one-shot script into a daemon, the rotation policy has to change with it, and that note is what tells the next person which assumption broke.

Finally, keep the log decision separate from the recovery decision. A failed probe should follow the connection's own recovery policy, not a rule hidden inside logrotate, and no logging arrangement should be the reason a working tunnel gets restarted.