The CronJob that quietly went dark
You run kubectl get cronjob and nothing looks wrong. SUSPEND reads False, ACTIVE reads 0. Then LAST SCHEDULE says 4d: no Job in four days. No failed pod, no CrashLoopBackOff, no warning event after the first hour. The controller stopped creating Jobs and never started again.
This is the missed-start deadlock, one of the few Kubernetes failures that is permanent without a human. Once it trips the schedule does not recover on its own, and by default nothing tells you.
How the controller decides whether to run
Counting missed start times
On each sync the CronJob controller looks at every scheduled time between the last Job it created and now, then acts on the most recent one. It walks forward from status.lastScheduleTime in schedule increments and counts the start times it steps over.
With startingDeadlineSeconds unset, that window has no floor: it runs from the last recorded schedule all the way to the current clock. The further behind the controller has fallen, the more start times land inside it. A */5 * * * * job books twelve per hour of downtime.
Where the number 100 comes from
The controller caps that count. Once it finds more than 100 missed start times, it abandons the calculation and logs a warning:
Cannot determine if job needs to be started: too many missed start times. Set or decrease .spec.startingDeadlineSeconds or check clock skew
The threshold lives in pkg/controller/cronjob/utils.go and has been 100 for years. The wording has drifted: older Kubernetes logged too many missed start times (> 100), current versions drop the number, so match on the phrase. The comment in the source is candid: a controller wedged "on Friday at 5:01pm" should let its hourly jobs catch up when someone restarts it on Tuesday, but a bug producing thousands of missed starts "would eat up all the CPU and memory of this controller," so the author "somewhat arbitrarily picked 100." The cap protects the controller, not your schedule.
What pushes a CronJob past 100
Control plane downtime
The obvious trigger is the controller being down: a managed control plane offline for nine hours during an upgrade, or a kube-controller-manager that crash-looped over a weekend. A one-minute CronJob racks up 540 missed starts in that time and comes back past the cap, unable to schedule again.
concurrencyPolicy Forbid plus a slow job
This one catches people out, because the cluster stays healthy. With concurrencyPolicy: Forbid, any scheduled time that arrives while the previous Job still runs is recorded as a missed start. A job that usually finishes in seconds but hangs for hours on a */1 schedule books one every minute it overruns, and enough of those trip the deadlock with no outage and no node pressure. A long suspend: true accrues missed starts the same way.
Why nothing pages you
The failure is silent by construction. The CronJob object still exists and reports SUSPEND: False, so a health check that lists CronJobs sees a healthy one. There is no Job, which means no pod and nothing for a pod-level alert to fire on. kubectl get cronjob shows a growing LAST SCHEDULE age, but only if a human runs it and reads that column.
Most alerts watch for the presence of a failure. This one produces no failing thing. It produces an absence.
Setting startingDeadlineSeconds, and its own trap
A deadline bounds the counting window
Setting startingDeadlineSeconds moves the floor of the window to now - startingDeadlineSeconds. The controller only counts starts missed inside that recent window, so the total can never climb toward 100, and the deadlock cannot form.
spec:
schedule: "*/5 * * * *"
startingDeadlineSeconds: 200
concurrencyPolicy: Forbid
The tradeoff you are accepting
The setting that prevents the deadlock also throws away catch-up runs. With startingDeadlineSeconds: 200 on a five-minute job, a controller down for an hour starts one Job on recovery, not twelve. For idempotent work that is fine; for a job that must run every interval it is not, and either way the missed intervals vanish unless something outside the cluster is counting.
Catching it from outside
Watch lastScheduleTime, not Job status
The cluster-side signal is status.lastScheduleTime. Alert when the gap against the schedule exceeds one interval plus a margin:
kubectl get cronjob nightly-report \
-o jsonpath='{.status.lastScheduleTime}'
kube-state-metrics exposes the same fact as kube_cronjob_status_last_schedule_time for Prometheus. Both depend on the very control plane that may be wedged, the weakness of every in-cluster check.
A success ping the cluster cannot fake
The check that survives a wedged controller lives outside it. Have the job confirm success to an external monitor, and alert on the ping that never arrives:
#!/bin/sh
set -euo pipefail
python generate_report.py
curl -fsS --retry 3 https://cronguard.app/api/ping/your-monitor-id
If the CronJob stops scheduling, the container never runs, the ping never fires, and the monitor alerts once the window closes. It knows nothing about missed-start counters, which is why it still works when the scheduler has quietly given up.
Frequently asked questions about the deadlock
Why does my Kubernetes CronJob stop creating Jobs with no error? The controller counts how many scheduled start times it missed since the last Job. Once that count passes 100 and startingDeadlineSeconds is not set, it stops scheduling and only logs a warning event that expires after about an hour. The object still reports SUSPEND False, so there is no failing pod and no obvious sign anything broke.
What does the "too many missed start times" message mean? It means the controller found more than 100 scheduled times between the last Job and now, and refused to enumerate them to avoid exhausting its own memory. Until you set or lower startingDeadlineSeconds, the job will not schedule again.
Does setting startingDeadlineSeconds fix the deadlock? Yes. It bounds the counting window to the last startingDeadlineSeconds, so the missed-start total can never reach 100. The tradeoff is that any run older than that deadline is skipped, so you trade a permanent stall for dropped catch-up runs.
Can a healthy cluster trigger this without any downtime? Yes. With concurrencyPolicy Forbid, every scheduled time that arrives while the previous Job is still running counts as a missed start. A frequent job that overruns for long enough can cross 100 while every node stays healthy, then never resumes.
How do I detect the deadlock before it causes damage? Watch status.lastScheduleTime and alert when the gap since the last schedule exceeds one interval plus a margin, using the kube_cronjob_status_last_schedule_time metric. Because that check leans on the same control plane that may be wedged, back it with an external dead man's switch expecting a success ping from each run.
Further reading
- Kubernetes Job Suspension and Why It Creates a Monitoring Blind Spot
- Kubernetes CronJobs vs Traditional Crontab: Key Differences
- Dead Man's Switch Monitoring: The Only Reliable Way to Watch Cron Jobs
Conclusion: The missed-start deadlock is not a bug. It is a deliberate guard that stops one wedged CronJob from taking the controller down with it. The cost is a schedule that stalls forever and pages no one, because the failure is the absence of a Job rather than a broken one. Set startingDeadlineSeconds so the deadlock cannot form, decide whether you want catch-up runs, and put an external monitor on the success signal so the day a schedule goes dark, something is still counting.
Sources: Kubernetes CronJob documentation, CronJob controller source, pkg/controller/cronjob/utils.go.
