How job suspension works in Kubernetes
A Kubernetes Job with spec.suspend: true exists in the cluster but produces no pods. The job controller sees it and does nothing until something sets that field to false.
On a standalone Job you create by hand, that is useful: it gives you time to inspect the object before any pods start. On a CronJob the effect is broader. Setting spec.suspend: true on the CronJob itself stops the controller from creating new Jobs for every schedule interval that passes. Jobs already running are not touched, but every missed interval adds up.
The catch is what happens when you unsuspend. If startingDeadlineSeconds is not set, Kubernetes immediately creates Jobs for all the intervals it missed. A CronJob suspended for an hour with a fifteen-minute schedule fires four Jobs at once when you re-enable it. That burst might be harmless. It might also hit a database, an external API, or a billing system four times in a row. Nothing warns you before it happens.
How Kueue manages suspension
Kueue is the Kubernetes SIG-batch project for batch job admission. Before any pods are created, it checks whether the cluster has enough capacity for the Job's resource requests.
The process runs without any action on your part. When you submit a Job to a Kueue-managed namespace with the kueue.x-k8s.io/queue-name label, Kueue's admission webhook intercepts it and sets spec.suspend: true before the Job is persisted, no matter what your manifest says. Kueue then creates an internal Workload object and checks the relevant ClusterQueue. If quota is available, it unsuspends the Job. If quota is exhausted, the Workload sits in the queue and the Job stays suspended with no timeout by default.
apiVersion: batch/v1
kind: Job
metadata:
name: nightly-report
labels:
kueue.x-k8s.io/queue-name: team-queue
spec:
template:
spec:
restartPolicy: Never
containers:
- name: reporter
image: myregistry/report:latest
resources:
requests:
cpu: "4"
memory: "8Gi"
A Job submitted like this gets suspended automatically while Kueue waits for enough CPU and memory quota. You can check the state with kubectl get workloads, but nothing fires an alert while the wait goes on. There is no built-in timeout that fails the job if it waits too long.
What changed in Kubernetes 1.36
Before Kubernetes 1.36, a suspended Job's resource requests were immutable. If a job was sitting in Kueue waiting for four GPUs that were never going to be available at the same time, the queue controller had two options: keep waiting, or delete the Job and recreate it with smaller requests. Deleting and recreating works, but it loses the Job's history, annotations, and the ownerReference back to the parent CronJob.
Kubernetes 1.36, released on April 22, 2026, promoted mutable pod resources for suspended Jobs from alpha to beta. A queue controller can now update resource requests and limits on a suspended Job, then set spec.suspend: false to resume it with the adjusted allocation, without deleting the object.
# Kueue reduces GPU request from 4 to 2, then resumes the job
spec:
suspend: false
template:
spec:
containers:
- name: trainer
resources:
requests:
nvidia.com/gpu: "2"
limits:
nvidia.com/gpu: "2"
For CronJob-spawned work, this means a job that would previously have been skipped under heavy cluster load can now be downsized to fit available quota and run. The Kubernetes project documentation frames this as letting CronJob instances progress with reduced resources rather than failing to run at all. What it does not do is tell the operator anything. Nothing in the default Kubernetes event stream records that the job ran with half its originally requested resources, or that it was suspended and resized before starting.
The gap in what Kubernetes surfaces
Kubernetes generates events for job state transitions. Those events expire after one hour by default. A job suspended for two hours before admission leaves no visible event trail by the time it finishes.
The CronJob controller tracks missed schedules, but stops counting after 100 missed runs. A CronJob suspended for a week with a ten-minute schedule interval stops recording missed runs long before the week is up.
kube-state-metrics exposes a kube_cronjob_missed_schedule_duration_seconds gauge that Prometheus can scrape and alert on. It requires kube-state-metrics, a Prometheus instance, and a configured alerting rule, none of which are set up by default. And even with all that in place, the gauge cannot tell you whether the job was suspended on purpose, held by Kueue waiting for quota, or missed because the CronJob controller itself was under load.
Setting startingDeadlineSeconds
startingDeadlineSeconds on a CronJob controls how late a job is allowed to start before that schedule slot is counted as missed.
spec:
schedule: "0 * * * *"
startingDeadlineSeconds: 300
suspend: false
With this set, a job that has not started within five minutes of its scheduled time is recorded as a missed run rather than left waiting for quota. It does not prevent the suspension itself, but it makes missed schedules visible in the CronJob's status and avoids the burst of simultaneous Jobs when you unsuspend.
The check that works for any failure mode
Suspension, quota waiting, pod eviction, process crash: from the outside they all look the same. The job did not complete on time.
The check that catches all of them is an external monitor watching for a success signal. The job pings an external endpoint when it finishes. If the ping does not arrive within the expected window, the job did not succeed. The monitor does not need to know anything about cluster state to reach that conclusion.
#!/bin/sh
set -e
# Do the batch work
python run_report.py
# Only executed on success
curl -fsS --retry 3 https://cronguard.app/api/ping/your-monitor-id
Set the window to the job's typical runtime plus the queue admission latency you see on your cluster. A job that finishes in ten minutes and normally waits two minutes in Kueue could reasonably have a thirty-minute window. The number comes from your SLA for the work, not from Kubernetes internals.
Frequently asked questions about Kubernetes job suspension
What does spec.suspend do to a Kubernetes Job? Setting spec.suspend to true creates the Job object in the cluster but prevents the job controller from creating any pods. No work runs until something sets the field back to false. Kueue uses this field as its admission mechanism, intercepting submitted Jobs and suspending them automatically until sufficient quota is available.
Does suspending a CronJob affect Jobs that are already running? No. Setting spec.suspend to true on a CronJob prevents the controller from creating new Jobs for upcoming schedule intervals, but Jobs the CronJob already created and any pods those Jobs have started continue running until they succeed or fail on their own terms.
What happens to missed schedule intervals when a CronJob is unsuspended? If startingDeadlineSeconds is not set, Kubernetes immediately creates Jobs for every interval that was missed while the CronJob was suspended, which can produce multiple simultaneous Jobs running at once. If startingDeadlineSeconds is configured, only intervals missed by less than that many seconds are rescheduled; older missed slots are counted as missed runs and dropped.
What did Kubernetes 1.36 change about suspended Jobs? Kubernetes 1.36 promoted the ability to update resource requests and limits on a suspended Job to beta. Before this, a Job's resource specifications were immutable once created, so adjusting them required deleting and recreating the Job and losing its history. Controllers like Kueue can now modify the CPU, memory, and GPU allocations of a suspended Job before resuming it, without deleting the object or losing its ownerReference back to the parent CronJob.
Why does an external monitor catch suspended Job failures when Kubernetes built-in tooling does not? Kubernetes job events expire after an hour by default, the missed schedule counter on CronJobs caps at 100, and kube-state-metrics gauges require additional tooling and alert rules to become actionable. An external monitor watching for a success ping does not care about cluster state at all. If the ping does not arrive, the job did not succeed, and that is equally true whether the job crashed, stayed suspended in Kueue, was resized and ran with reduced resources, or never started because the schedule was missed.
Further reading
- Kubernetes CronJobs vs Traditional Crontab: Key Differences
- Dead Man's Switch Monitoring: The Only Reliable Way to Watch Cron Jobs
- Cron Job Observability: Metrics, Logs, and Traces for Scheduled Tasks
Conclusion: Kubernetes handles Job suspension correctly — holding work until quota is available is better than letting pods fail immediately, and Kubernetes 1.36 makes that smarter by letting controllers downsize resource requests before resuming a Job. But correct design is not the same as visible behavior. A Job suspended for three hours looks identical to one that has been waiting three seconds, and neither produces an alert by default. The answer is the same as it is for cron: an external monitor that watches for the presence of success, not the absence of error.
Sources: Kubernetes CronJob documentation, Kubernetes v1.36: Mutable Pod Resources for Suspended Jobs, Kueue documentation.
