A Job that says Running and has no pods
On 16 September 2026 someone filed Kubernetes issue #142142. They had deleted a Job, created a new one with the same name straight away, and the new one did nothing. kubectl get job reported it as Running, 0/1 completions, 18 minutes old. Zero pods of any kind. Nothing had failed, so nothing alerted.
The trigger was ordinary. The cluster ran RKE2 v1.35.7, whose helm-controller deletes and recreates its helm-install-* Jobs when a chart's configuration changes. In a controlled reproduction the reporter watched one Job stay wedged for 13 minutes and 9 seconds before stepping in.
The fix merged on 24 September and was backported to three branches. Kubernetes v1.35.10, v1.36.6 and v1.37.2 carry it, all published on 28 October. Anything older on those lines still has the bug.
How the Job controller lost track
The Job controller keeps an in-memory ledger called expectations. When it asks the API server for a pod, it writes down that it expects one creation, and it crosses that out once the informer reports the pod. While an expectation is open, it creates no pods, so it never doubles up.
The ledger is keyed by namespace and name. The UID plays no part.
The race, step by step
- The controller starts creating a pod for Job
reportand records an expectation underjobs/report. - Someone deletes
reportwhile that creation is in flight. - A new Job called
reportappears with a new UID. - The controller syncs it, finds the old expectation under
jobs/reportstill open, and skips pod management.
In the controller-manager log you get a status write rejected because the UID in the precondition does not match the object. That is the only error.
What the fix changes
PR #142247 adds one call to the Job delete handler, jm.expectations.DeleteExpectations(logger, key), so a replacement with the same name starts with a clean ledger. The PR says the old code left the replacement alone "until another event triggers it or expectations timeout". In the issue that meant 18 minutes once, 13 another time.
Your CronJobs are probably fine
How CronJob names its Jobs
The CronJob controller names each Job after the CronJob plus a hash of the scheduled time. Two scheduled runs never share a name, so the runs it creates itself cannot hit this race.
Where fixed names come from
The risk sits in the scheduling people bolt on around Kubernetes. A CI schedule that runs kubectl delete job nightly-report and then kubectl create job nightly-report --from=cronjob/report reuses a name every night. So does a retry wrapper that deletes a slow Job and reapplies the same manifest, and so does the RKE2 and K3s helm-controller.
It only bites when the old Job was still creating pods at the moment of deletion, so a tight retry loop around a Job that just started is the worst case.
Stop reusing Job names
Upgrade the control plane first
Check the server version with kubectl version and move to v1.35.10, v1.36.6 or v1.37.2 or later. On EKS, GKE or AKS that depends on when your provider rolls the patch.
Put something unique in every name
Patching fixes this bug. Unique names also cover the next cache that happens to be keyed by name:
kubectl create job "nightly-report-$(date +%Y%m%d%H%M)" --from=cronjob/report -n reporting
For manifests, use generateName with kubectl create instead of a fixed name with kubectl apply:
apiVersion: batch/v1
kind: Job
metadata:
generateName: nightly-report-
namespace: reporting
spec:
ttlSecondsAfterFinished: 86400
backoffLimit: 2
template:
spec:
restartPolicy: Never
containers:
- name: report
image: registry.example.com/report:1.4.2
command: ["/app/run-report"]
ttlSecondsAfterFinished removes finished Jobs, so the unique names do not pile up.
Unsticking a Job that is already wedged
Don't wait it out. Delete the stuck Job and create its replacement under a new name. Restarting kube-controller-manager also clears the ledger, since it only lives in memory, but on a managed control plane that process isn't yours to restart.
Finding Jobs that never started
A Job caught by this race looks like this: not suspended, no Complete or Failed condition, and no pods counted, long after it was created. This lists every such Job older than ten minutes:
kubectl get jobs -A -o json | jq -r --argjson now "$(date +%s)" '
.items[]
| select(.spec.suspend != true)
| select([.status.conditions[]? | select(.status == "True" and (.type == "Complete" or .type == "Failed"))] | length == 0)
| select(((.status.active // 0) + (.status.succeeded // 0) + (.status.failed // 0)) == 0)
| select(($now - (.metadata.creationTimestamp | fromdateiso8601)) > 600)
| "\(.metadata.namespace)/\(.metadata.name)"'
If you can read the controller-manager logs, search for "Precondition failed: UID in precondition". On managed Kubernetes you usually can't.
Why a check-in still matters
That query only sees Jobs that exist. If the CI runner was down, or the retry wrapper crashed between delete and create, there is no Job to find.
A check-in at the end of the Job's command covers that case as well. No work, no ping:
#!/bin/sh
set -eu
/app/run-report
curl -fsS --retry 3 https://www.cronguard.app/api/ping/nightly-report-id > /dev/null
Set the grace period to the latest the report can finish before someone downstream notices. Wedged or never created, the monitor only sees a missing ping.
Frequently asked questions about Kubernetes Jobs stuck at zero pods
Why does a recreated Kubernetes Job sometimes create no pods? The Job controller tracks pending pod creations in an in-memory ledger keyed by namespace and name, not by UID. If a Job is deleted while a pod creation is still in flight and a new Job with the same name is created straight away, the new Job inherits the old open expectation and the controller skips creating its pods.
Which Kubernetes versions fix the stuck Job bug? The fix from PR 142247 shipped in Kubernetes v1.35.10, v1.36.6 and v1.37.2, all published on 28 October 2026. Earlier patch releases of those minor versions still have the bug.
Are Kubernetes CronJobs affected by this race? The runs a CronJob creates itself are not, because each Job is named after the CronJob plus a hash of the scheduled time. The risk comes from tooling that deletes and recreates a Job under a fixed name, such as CI scripts, retry wrappers and the RKE2 helm-controller.
How do I get a wedged Job running again? Delete it and create the replacement under a new name. Restarting kube-controller-manager also clears the stale expectation, but on managed Kubernetes you do not control that process, so a new name is the practical option.
How can I detect a Job that never started? Query for Jobs that are not suspended, have no Complete or Failed condition and have no pods counted long after creation. Add a check-in at the end of the Job's command too, because that also catches runs that were never created.
Further reading
- The Kubernetes CronJob Deadlock That Stops Your Schedule For Good
- Kubernetes Job Suspension and Why It Creates a Monitoring Blind Spot
- Dead Man's Switch Monitoring: The Only Reliable Way to Watch Cron Jobs
Conclusion: Before v1.35.10, v1.36.6 and v1.37.2, a Job deleted and recreated under the same name could sit at zero pods for a quarter of an hour without a single failure condition. Your own CronJob runs are safe from it. The scripts and controllers that recycle Job names are not, and they are the place to put unique names and an outside check-in.
Sources: Kubernetes issue 142142, Kubernetes PR 142247, Kubernetes v1.37 changelog, Kubernetes Jobs documentation.