The successor kept the silence
On most modern Linux distributions the nightly backup no longer lives in a crontab. It lives in a .timer unit paired with a .service unit. Maintainers moved there for real logging, dependency ordering, and a systemctl status you can read.
None of that tells you when the job stops running. A systemd timer fails as quietly as the cron line it replaced. In one respect it used to fail worse.
How a timer actually fails
A timer is two units. The .timer decides when, the .service decides what, and that split is where the trouble hides.
The service fails, the timer keeps firing
When a oneshot service exits non-zero, systemd marks that unit failed and moves on. The timer does not care. It stays active and schedules the next run exactly as before.
systemctl status backup.service
# Active: failed (Result: exit-code) since Thu 2026-08-27 03:00:12 UTC
You only see that line if you go looking. The next run still shows up as scheduled, so everything looks fine while the job has errored out every night for a week.
Persistent catch-up hides missed windows
An OnCalendar= timer can set Persistent=true. Per the man page, systemd stores the last trigger time on disk and, on activation, runs the service immediately if it would have fired while the timer was inactive. Anacron behaviour: a laptop asleep at 03:00 runs the backup when it wakes.
Useful, and also a blind spot. A box that reboots at the wrong minute, catches up an hour late, and succeeds looks identical to one that ran on time. Whether it landed in the window you needed is a question the timer never asks.
Reading the evidence
list-timers shows intent, not health
systemctl list-timers is the first thing to reach for. It shows the next elapse, the time left, the last run, and which unit fires.
systemctl list-timers --all
# NEXT LEFT LAST PASSED UNIT ACTIVATES
# Fri 2026-08-28 03:00:00 UTC 14h left Thu 2026-08-27 03:00:00 UTC 9h ago backup.timer backup.service
LAST is when the timer fired, not when the service succeeded. A timer that kicks off a service that fails every night still shows a recent LAST. The column that would tell you the truth is not there.
The journal, and where it used to go
The real record is in the journal, per unit:
journalctl -u backup.service --since "3 days ago"
This is where a systemd job beats a bare crontab line. Output is captured, timestamped, and attributed to the unit, no shell redirection. Except that for years the journal did not keep it. The default Storage= was auto, which meant persistent only when /var/log/journal already existed. On minimal server images it did not, so the journal lived in /run and vanished on reboot. A timer job fails, the box reboots hours later, and the command that would explain it returns nothing.
What systemd 259 changed
systemd 259, released on 17 December 2025, changed the default. From the release notes: the default storage mode for the journal is now persistent, where previously it was auto and the presence of /var/log/journal decided the outcome. New installs keep logs across reboots.
Good change. Not retroactive. Existing hosts keep whatever journald.conf they were built with, and a running fleet is mostly existing hosts. Check the box you care about instead of assuming the new default reached it:
journalctl --disk-usage
# a tiny number here means your history is probably still volatile
If it is still volatile, set Storage=persistent in /etc/systemd/journald.conf, create /var/log/journal, and restart systemd-journald.
The built-in failure hook, and its limits
systemd does offer a hook. Any unit can name an OnFailure= unit that starts when it enters the failed state, the usual way people wire a timer to an alert.
[Unit]
Description=Nightly database backup
OnFailure=notify-failure@%n.service
What OnFailure catches
Exactly one case: the service ran and exited non-zero. Your script returns an error code, the handler starts, you get your page.
What it never sees
It says nothing when the service never ran at all. A timer masked during a config push, a unit disabled by a bad deploy, a clock that jumped so OnCalendar= drifted, a dependency that blocked the start: none of those trigger OnFailure=, because the failure is the absence of a run. You also own the handler unit on every machine. That is the same wall cron hit: watching for a failure signal only works when a process is alive to send it.
A dead man's switch for a timer
The reliable pattern flips the question: watch for the absence of success, not for failure. Have the job report in only when it finishes cleanly, to an external monitor that alerts when the report does not arrive on time.
With a oneshot service this drops in cleanly: ExecStartPost= runs only after ExecStart= exits successfully.
[Service]
Type=oneshot
ExecStart=/usr/local/bin/backup.sh
ExecStartPost=/usr/bin/curl -fsS --retry 3 https://cronguard.app/api/ping/your-monitor-id
If the backup fails, ExecStartPost= never runs and the check-in never arrives, so the monitor alerts. This catches every mode at once: the errored service, the masked timer, the powered-off host, the drifted clock. The monitor knows only that 03:00 came and no success did, the signal you wanted the whole time.
Frequently asked questions about monitoring systemd timers
Does a systemd timer alert me when its job fails? No. systemd marks the unit failed and the timer schedules the next run anyway, so the error sits in the unit state until someone runs systemctl status or reads the journal. The schedule looks healthy while every run breaks.
What does Persistent=true do when my machine was off? On an OnCalendar timer it runs the service on the next activation if the timer would have fired while the system was down. It is anacron-style catch-up, so a late run looks identical to an on-time one and can hide that the job missed the window you cared about.
Why were my timer's logs gone after a reboot? The journal used to default to auto storage, persistent only when /var/log/journal already existed. On minimal server images it often did not, so the journal lived in memory and was wiped on reboot. systemd 259 changed the default to persistent, but existing hosts keep their old config.
How is OnFailure different from real monitoring? OnFailure only fires when the service actually ran and exited non-zero. It cannot fire when the run never happens, so a masked timer, a disabled unit, a powered-off host, or a drifted clock all slip past it. External monitoring alerts on the absence of a check-in, which covers those.
How do I add a dead man's switch to a systemd timer? On a Type=oneshot service, put the check-in in ExecStartPost with a curl to a ping URL, since ExecStartPost runs only after ExecStart succeeds. If the job fails or never runs, the check-in never arrives and the monitor alerts once the window passes.
Further reading
- Cron Alternatives Compared: systemd Timers, Celery Beat, and More
- Everything Runs at Midnight: Spreading Out Your Scheduled Jobs
- Dead Man's Switch Monitoring: The Only Reliable Way to Watch Cron Jobs
Conclusion: systemd timers fixed cron's logging and dependency problems and left the core one untouched. A job that stops running still tells no one. The unit state and the journal hold the evidence, and systemd 259 finally keeps it across a reboot, but both only help someone who thinks to look. The signal that scales is an external monitor watching for the success that should have arrived and did not.
Sources: systemd.timer(5), What's new in systemd v259 (LWN).
