A Five-Minute Check That Ran Every Five Hours
On 30 October 2026 the API behind textstack, an open-source project on GitHub, returned 502 for 44 minutes. There was a health check for exactly this case: a GitHub Actions workflow with cron: '*/5 * * * *'. Its last run before the outage was five hours earlier. A person found the outage by looking at a red cross in CI.
The maintainer pulled the real run times for 29 and 30 October: 06:46, 13:44, 19:13, 23:16, 02:37, 08:56. Three to six hours apart. The fix they merged left the cron line alone and put a comment above it that opens with: "Five minutes is what this asks for, not what it gets."
Nobody was watching the watcher. That is the whole story.
Your Monitor Inherits Its Scheduler's Failures
A health check on a schedule is a cron job. Anything that makes cron jobs late or makes them vanish hits the check too, and the check cannot report on itself.
The Shared Platform Problem
GitHub's docs say the schedule event "can be delayed during periods of high loads," that "high load times include the start of every hour," and that "some queued jobs may be dropped." The textstack issue adds the part that matters here: GitHub "does not queue what it drops." A dropped run never shows up late. It just never happens.
Other people saw the same thing this autumn. A community thread opened on 2 October 2026 logged scheduled runs created more than five hours late on 30 September, six hours late on 1 October, and none at all by the morning of 2 October. Manual dispatch worked the whole time. In the author's words: "The delay is in run creation, not hours waiting for a runner."
The Shared Host Problem
So move the check onto the production server as a systemd timer. The cadence is fixed, and now the check lives on the box it is checking. When that box loses power or fills its disk, the check dies with it. A Kubernetes CronJob probing its own cluster has the same flaw one level up.
Detection Time Is Not the Check Interval
The interval in the config is a request. Worst-case detection is the largest real gap between runs, multiplied by however many consecutive failures your alert rule wants, plus the time it takes a page to reach someone.
Consecutive-Failure Rules Multiply the Gap
"Alert after three failed checks" is fine at five minutes: roughly fifteen minutes to page. At three to six hours per run, the same rule can take most of a day. A 44-minute outage never gets close.
Measure the Cadence You Actually Get
On GitHub Actions the run history has the numbers. Filter to scheduled runs so manual reruns do not paper over the gaps:
gh run list --workflow health-check.yml --event schedule \
--limit 100 --json createdAt --jq '.[].createdAt' | sort
Subtract neighbouring timestamps and look at the biggest gap. Ignore the average. On a host, systemctl list-timers shows the last and next trigger, and the service's journal shows when it really ran.
A dashboard that shows the last result has the same blind spot. Green stays green when no new run arrives, and a rule like "the last two runs failed" only looks at runs that exist.
Check the Body, Not the Status Code
During the textstack outage the front page kept answering 200, because nginx served the static build from disk. Only the API was down. A check on the home page would have passed. The maintainer's follow-up is blunt: a prober has to hit /api/health/ready, and "it has to read the body, not just the status code."
#!/bin/bash
set -euo pipefail
body="$(curl -fsS --max-time 10 https://example.com/api/health/ready)"
grep -q '"status":"ok"' <<<"$body"
curl -f fails on an HTTP error. The grep fails when the endpoint answers 200 and still says it is unhealthy. Either way the script exits non-zero.
Put the Prober in a Different Failure Domain
The textstack comment says it outright: nothing hosted on GitHub can see the site go down "when GitHub is the unreliable part." The options in the issue were an external prober, a systemd timer on the server, or both. Pick both. Each covers the gap the other leaves.
Make the Monitor Check In Too
Run the check from a timer on the server, where the schedule holds, and have it report success somewhere off the server:
# /etc/systemd/system/api-health.timer
[Timer]
OnCalendar=*:0/5
Persistent=true
[Install]
WantedBy=timers.target
#!/bin/bash
set -euo pipefail
body="$(curl -fsS --max-time 10 https://example.com/api/health/ready)"
grep -q '"status":"ok"' <<<"$body"
# Only reached if the API is healthy
curl -fsS --retry 3 https://cronguard.app/api/ping/your-monitor-id
The outside side needs one rule: no ping in six or seven minutes, alert. It fires on an unhealthy API. It also fires when the timer stops or the host goes away, because silence is the signal.
Write Down the Cadence You Observed
Copy this part even if you change nothing else. textstack kept */5, since a dropped run costs nothing and the check takes six seconds. The comment above it now records what was measured, when, and what it cost. The issue puts it well: "a schedule that lies in a file is worse than no schedule."
Frequently asked questions about health checks on a schedule
Why did a five-minute health check miss a 44-minute outage? It ran as a GitHub Actions scheduled workflow, and GitHub delays scheduled runs under load and drops some without queuing them. The textstack check really ran three to six hours apart, and its last run before the outage was five hours earlier.
Is moving the health check onto the production server enough? It fixes the cadence but puts the check on the same host as the service. If that host loses power or fills its disk, the check stops too and reports nothing. Pair the on-host timer with an outside watcher that expects a check-in.
How do I work out my real detection time? Take the largest gap between actual runs, not the configured interval, and multiply it by the number of consecutive failures your alert rule needs. Then add the time a page takes to reach someone. On a check that runs every few hours, a three-failure rule can take most of a day.
Why should a health check read the response body? In the textstack outage the front page kept returning 200 while the API returned 502, so a check on the home page would have passed. Probing a readiness endpoint and checking that the body reports a healthy status catches failures a status code hides.
How do I get alerted when the health check itself stops running? Have the check ping an outside service only after it passes, and alert when no ping arrives within the expected window. That rule fires on an unhealthy service and also when the timer stops or the host disappears.
Further reading
- Why Your GitHub Actions Scheduled Workflows Run Hours Late
- Dead Man's Switch Monitoring: The Only Reliable Way to Watch Cron Jobs
- Monitoring systemd Timers: The Cron Successor Fails Just as Quietly
Conclusion: The textstack check was correct and cheap. Its schedule was the problem, and nobody had measured it. Pull the real run times for every check you rely on, and make sure something outside its host notices when it goes quiet.
Sources: GitHub Docs: Events that trigger workflows, textstack pull request 1064, textstack issue 1054, GitHub community discussion 209332, systemd.time(7).
