Four hours of cron runs that never happened
At 17:03 UTC on 20 October 2026, Railway paused new builds for most workspaces. A fix went out at 20:20 and the incident closed at 20:56. The status page had one line about scheduled work: "Cron jobs scheduled during the pause did not run, and will continue from their next scheduled time."
That is honest, and it is the default almost everywhere. A run that was due while the scheduler could not start it is gone. The next one fires on time as if nothing happened.
Picture an hourly export of "the last 60 minutes". Four runs skipped, four hours of orders never exported. No error either, since no job ran to raise one.
Who replays a missed run
Fewer schedulers than you would hope.
Vixie cron and cronie skip it
Cron checks its tables once a minute and runs whatever matches. The cronie cron(8) man page documents one catch-up case: a forward clock jump of less than three hours, as with daylight saving time, runs the skipped jobs immediately. A stopped daemon or a powered-off host gets nothing.
anacron counts in days
anacron ships with cronie and exists for exactly this gap. Its man page says it "does not assume that the machine is running continuously." It checks whether each job ran in the last n days and, if it did not, runs it after a delay. Because it stores the date and not the hour, it can rescue a missed daily job. A missed hourly one is out of reach.
systemd timers with Persistent=true
With Persistent=true, systemd writes the last trigger time to disk. On activation it triggers the service immediately "if it would have been triggered at least once during the time when the timer was inactive." The setting works only on OnCalendar= timers, defaults to false, and the catch-up still waits for RandomizedDelaySec=.
[Timer]
OnCalendar=*-*-* *:00:00
Persistent=true
Note the words "at least once". Four missed hourly runs produce a single activation.
Kubernetes CronJobs and startingDeadlineSeconds
The CronJob controller counts the schedules it missed. Set startingDeadlineSeconds and any run that could not start inside the deadline is skipped, while later runs carry on. Leave it unset and long controller downtime can push the count past 100, at which point the CronJob stops scheduling for good. In both cases recovery creates one Job for the most recent missed time.
spec:
schedule: "0 * * * *"
startingDeadlineSeconds: 3600
Hosted schedulers drop by design
Railway's cron docs already say a run is skipped while the previous one is still active. The October incident added another case: a platform pause skips runs too. GitHub Actions is blunter. The schedule event can be delayed under load, and "some queued jobs may be dropped."
One catch-up run is not every missed run
The schedulers that do catch up behave the same way. They spot the gap and start the job once.
For log rotation or certificate renewal, one run is enough. It is wrong when the unit of work is the interval itself. Trigger that hourly order export once after a four-hour gap and it exports one hour. Three hours stay behind. The catch-up run exits 0, a monitor that only looks at the latest run is happy, and the hole now hides behind a success.
The scheduler cannot fix this. The job has to.
Write jobs that backfill themselves
Process everything since the last success
Keep a watermark: the end of the last range the job finished. Each run processes from the watermark up to now and moves the watermark only once the work succeeds. A missed run then just means the next run has more to do.
#!/bin/bash
set -euo pipefail
STATE=/var/lib/export-orders/last_success
SINCE="$(cat "$STATE" 2>/dev/null || date -u -d '1 hour ago' +%FT%TZ)"
UNTIL="$(date -u +%FT%TZ)"
/usr/local/bin/export-orders --since "$SINCE" --until "$UNTIL"
echo "$UNTIL" > "$STATE.tmp" && mv "$STATE.tmp" "$STATE"
curl -fsS --retry 3 https://cronguard.app/api/ping/your-monitor-id
A job that produces one artefact per period, say a daily report, should loop over every period since the last one it completed:
LAST="$(cat /var/lib/daily-report/last_date)"
TODAY="$(date -u +%F)"
d="$(date -u -d "$LAST + 1 day" +%F)"
while [[ "$d" < "$TODAY" ]]; do
/usr/local/bin/daily-report --date "$d"
echo "$d" > /var/lib/daily-report/last_date
d="$(date -u -d "$d + 1 day" +%F)"
done
Make the replay safe to run twice
Sometimes a watermark job reprocesses a range: it crashed before the watermark write, or someone backfilled by hand. Key the output on the period so a second pass overwrites the first. An upsert on report_date does it, and so does a file or object key that contains the date. Once a rerun is harmless, catching up stops being scary.
Detecting the hole the outage left
Alert on the missing ping, with a grace period
A dead man's switch catches exactly what the platform did not report: a run that should have checked in and never did. Give an hourly job a sensible grace period and a four-hour pause pages you within the first hour, whatever the status page says.
Reconcile after every platform incident
When your provider posts an incident, hold its window up against your schedules. Read last-run times with systemctl list-timers --all or the LAST SCHEDULE column of kubectl get cronjob. Check that each watermark has moved past the end of the incident. For the other jobs, decide yourself whether a manual run is needed.
Frequently asked questions about missed cron runs
Does cron run jobs it missed while the server was down? No. Cron and cronie only evaluate the minutes they are running for, so runs due while the daemon or host was down are skipped. The only built-in catch-up in cronie covers forward clock jumps of under three hours.
Does systemd Persistent=true replay every missed run? No. It triggers the service once on activation if the timer would have fired at least once while it was inactive. A timer that missed four hourly runs gets one activation, and the setting only works on OnCalendar timers.
What happened to Railway cron jobs during the October 2026 incident? Railway paused builds from 17:03 to 20:56 UTC on 20 October 2026. Its status page said cron jobs scheduled during the pause did not run and would continue from their next scheduled time, so those runs were never queued.
How do I make a scheduled job survive a missed run? Have it process everything since its last successful run, not a fixed window. Store a watermark, move it only after the work succeeds, and key the output on the period so a rerun overwrites the earlier result.
How can I tell which runs an outage skipped? Use a dead man's switch that expects a ping from every run, so a missing check-in alerts you during the outage. Afterwards, compare the incident window with your last-run times and confirm each watermark has moved past it.
Further reading
- The Kubernetes CronJob Deadlock That Stops Your Schedule For Good
- Monitoring systemd Timers: The Cron Successor Fails Just as Quietly
- Dead Man's Switch Monitoring: The Only Reliable Way to Watch Cron Jobs
Conclusion: Assume your scheduler will lose runs, because it will. The best of them replay a gap once and the hosted ones say outright that they drop work. Railway's four-hour pause was a routine incident, and that is the point: the job has to know where it left off, a rerun has to be harmless, and something outside the platform has to notice when a check-in does not arrive.
Sources: Railway incident, Railway cron docs, cron(8), anacron(8), systemd.timer(5), Kubernetes CronJob docs, GitHub Actions events.
