Green on the dashboard, nothing on disk
On 20 October 2026 Jepsen published its analysis of Orkes Conductor, the workflow orchestrator that started life at Netflix. One finding should bother anyone who runs scheduled work through an orchestrator: after a database failure, Conductor could mark a workflow COMPLETED while its task never executed.
Teams move nightly jobs off crontab for the dashboard. Green means done. Here it meant nothing ran.
What Jepsen tested
Kyle Kingsbury tested Orkes Conductor 5.3.2, 5.4.0 and the 5.5.0 release candidates, backed by Postgres and Redis. The workload was one workflow with one task that logged every execution. The faults were ordinary: process pauses, kills and network partitions.
Conductor promises at-least-once delivery and that workflows eventually terminate. Jepsen checked that against what the workers logged, and treated the API's answer as the thing under test. Steal that method.
Three findings that hide a missed run
The report lists nine issues. In three of them the orchestrator's view and reality drift apart and nothing raises an error.
Completed without running
Issue #9. To retry a timed-out task, the sweeper made two Postgres writes: insert a new task record, then mark the old one superseded. If Postgres failed in between, Conductor ignored the failed insert. Every task now looked terminal, so the workflow became COMPLETED. Fixed in 5.5.6.
Running forever
Issue #5. Starting a workflow writes to Postgres, then enqueues it. A crash in between left a workflow in the database but not in the queue. The sweeper only reads the queue, so it sat in RUNNING forever. Since 5.5.0 a background process re-enqueues abandoned executions.
Gone entirely
Issue #7. After a partition, a server could commit the workflow, fail to enqueue it, then delete its own record to undo the write. Lookups returned nothing for good. Also fixed in 5.5.0.
The hour-long stall that works as designed
Orkes does not count issue #2 as a bug. After a Redis or Postgres fault, a task could sit IN_PROGRESS on a worker that never started it, until the response timeout fired. Jepsen measured that default at one hour. It breaks no stated guarantee, so Orkes left it.
Which timeout default applies to you
The sources disagree. Jepsen gives the default responseTimeoutSeconds as one hour; the Conductor OSS docs say 600 seconds. Trust neither and set it explicitly:
{
"name": "nightly_ledger_export",
"retryCount": 3,
"timeoutSeconds": 1800,
"responseTimeoutSeconds": 120,
"timeoutPolicy": "RETRY"
}
Short works if the task sends status updates while it runs. A task that goes quiet for twenty minutes needs more room.
Two workers, one task
Leases cannot give you mutual exclusion
Issue #1 needed no faults at all. Two workers ran the same task at the same time. Conductor guards tasks with a lock lease, one minute by default via conductor.workflow.lockLeaseTime. With defaults Jepsen saw this about once every six hours; with a one-second lease, every few minutes. A sweeper race behind it was fixed in 5.5.6. The rest you cannot fix: on an asynchronous network a lease does not guarantee a single holder.
Make the task safe to run twice
Jepsen's advice is that tasks should be idempotent and commute. For a scheduled export, key the side effect on the logical run, not on the execution:
def export_ledger(run_date):
rows = build_export(run_date)
db.execute(
"""
INSERT INTO ledger_exports (run_date, row_count, checksum)
VALUES (%s, %s, %s)
ON CONFLICT (run_date) DO NOTHING
""",
(run_date, len(rows), checksum(rows)),
)
Let two workers run it. One insert lands, the other does nothing.
Scheduled workflows inherit all of this
Conductor's scheduler takes Spring six-field cron expressions, seconds first, with zoneId defaulting to UTC. Scheduled runs go through the same start path Jepsen broke.
What the scheduler does after downtime
runCatchupScheduleInstances defaults to false. After an outage the scheduler resumes from the current time and missed slots are skipped. Turn it on and you get a burst of catch-up runs.
No overlap policy
The Orkes docs say there is no native overlap policy. A run stuck in RUNNING for an hour does not stop the next slot from starting another. Idempotency again.
Watching the work instead of the status
Check in after the side effect
Report success from the code that did the work, after the work is durable. The orchestrator's status is the thing that lied, so polling it proves nothing:
curl -fsS --retry 3 https://cronguard.app/api/ping/ledger-export
Run that from the task worker after the insert commits. A workflow marked COMPLETED with no task, a workflow stuck in RUNNING, and a workflow that vanished all look the same from outside: no ping. That is what a dead man's switch watches for.
Size the grace window to the stall
Set the monitor's grace period above the job's normal runtime plus the response timeout you configured. Leave the one-hour default alone and a stalled task can still finish legitimately an hour late. Lower the timeout or live with a wide window.
What to do this week
Upgrade Orkes Conductor to 5.5.6, as Jepsen recommends. Conductor OSS was not tested, though Jepsen expects it leans on its storage just as hard.
Frequently asked questions about orchestrator status and missed runs
What did Jepsen find in Orkes Conductor? The October 2026 analysis found nine issues, including workflows marked COMPLETED whose task never ran, workflows stuck in RUNNING forever, and workflows that disappeared. It also saw two workers execute the same task at once.
How can a workflow be COMPLETED if its task never ran? To retry a timed-out task, Conductor wrote a new task record and then marked the old one superseded. If Postgres failed between those writes, the failure was ignored and the workflow was marked COMPLETED. Orkes fixed this in 5.5.6.
Which Conductor version should I run? Jepsen recommends 5.5.6. The abandoned and lost workflow bugs were fixed in 5.5.0, and the never-executed completed workflows in 5.5.6.
Why can the same task run on two workers at once? A lock lease cannot guarantee mutual exclusion on an asynchronous network. Jepsen saw it about once every six hours with default settings, so key each scheduled task's writes on the logical run date.
How do I detect a scheduled workflow that silently did not run? Send a check-in from the task code after its side effect commits, and alert when it does not arrive. A wrongly COMPLETED, stuck or lost workflow produces no check-in.
Further reading
- Durable Job Queues Are Replacing Cron: New Failure Modes
- Dead Man's Switch Monitoring: The Only Reliable Way to Watch Cron Jobs
- Preventing Duplicate Cron Jobs When You Scale to Multiple Servers
Conclusion: An orchestrator's status is its own belief about what happened, and it lives in the same Postgres and Redis that Jepsen kept killing. Conductor has fixed the worst of it in 5.5.6. The next orchestrator will have its own version of issue #9, so the proof that a scheduled run happened should come from the job.
Sources: Jepsen: Orkes Conductor 5.3.2, Orkes Conductor: Scheduling Workflows, Conductor OSS task definition reference.
