Back to blog

One Tenant's 01:00 Burst Stalled Atlassian's Shared Job Scheduler

On 1 October 2026 one consumer of Atlassian's shared scheduler queued a surge of tasks for 01:00 UTC. Pipelines, triggers and reminders were dropped or ran late.

CronGuard TeamCron Job Monitoring Experts
6 min read
A black-and-white photo of a laptop at the right edge of a desk, its screen filled with a terminal system monitor showing memory bars, network graphs and a process list, next to a blurred bright window

What broke at 01:00 UTC on 1 October

At 01:00 UTC on 1 October 2026, scheduled work across Atlassian Cloud stopped running on time. The postmortem gives the cause in one line: "One of the platform consumers scheduled an abnormally high volume of tasks to be executed at the same time." That order-of-magnitude surge exceeded the platform's architectural and scaling limits.

Jira, Jira Service Management, Confluence, Bitbucket Cloud and Forge apps all hand their timed work to that one platform. So one consumer's burst became everyone's backlog until 07:25 UTC.

Atlassian did not say what the tasks were. Look at the timestamp, though. First hour of a new month, first hour of a new quarter. That is where people put schedules.

One root cause, four kinds of failure

Each product failed differently, and each failure needs its own detector.

Bitbucket scheduled pipelines did not run

"Scheduled pipelines due between 01:00 and about 03:50 UTC did not run." Nothing was queued. The advice was to wait for the next run or trigger one by hand.

Forge scheduled triggers were not replayed

Forge triggers "stopped at 01:00 UTC and resumed from about 04:00 UTC; missed invocations were not replayed." Three hours of invocations were gone.

Jira notifications ran late

Filter subscriptions and automation rules triggered by SLAs or time in status ran late. They did run, hours after they were due.

Confluence reminders were never sent

Some calendar reminders due between 01:00 and about 03:45 UTC were not sent at all.

Late and dropped need different alerts

At 01:05 a dropped job and a late job look the same: nothing has checked in.

A dead man's switch catches both, provided the grace period is set per job. Ask how late the job may run before someone downstream gets hurt. A nightly export feeding a 09:00 report can slip two hours. A pipeline that rotates a short-lived credential can't. Use that number as the grace period.

The alert's history then tells you which case you had. A late run checks in eventually and the alert clears by itself. A dropped run never checks in, so the alert stays open until the next successful run. Cleared at 04:10 means late. Open until the next night means dropped, and there is a gap to backfill.

In a Bitbucket schedule the check-in goes last, so it only fires when the work succeeded:

pipelines:
  custom:
    nightly-export:
      - step:
          name: Export orders
          script:
            - ./scripts/export-orders.sh
            - curl -fsS --retry 3 https://cronguard.app/api/ping/your-monitor-id

Hosted schedulers are shared schedulers

Your job on GitHub Actions, Cloudflare or Bitbucket sits in a queue with jobs you will never see. A sensible schedule can still miss because another tenant's was not sensible.

You can stop waiting for the status page, though. Atlassian's first update went out at 02:55 UTC, nearly two hours after work stopped. A check-in monitor on an hourly job would have paged first.

When the shared scheduler is yours

Many companies run the same shape in-house: one Kubernetes cluster where every team ships CronJobs, or one Celery Beat for everything. Atlassian's remedial actions work as a checklist there: fair scheduling guardrails, a fast way to throttle or pause scheduled workloads, and more capacity.

Separate lanes per team

Put each team in its own Kubernetes namespace so quotas apply per tenant. In Celery, give each team its own queue and workers. A flood then waits in its own queue.

Cap the Jobs one namespace can hold

Object count quotas limit Jobs and CronJobs per namespace:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: batch-limits
  namespace: team-reports
spec:
  hard:
    count/jobs.batch: "50"
    count/cronjobs.batch: "20"

There is a price. At the limit the API server refuses the next Job and that run never happens. Size the quota above normal peak and watch that namespace for missing runs.

A pause switch for one namespace

When one team floods the cluster, suspend that team's CronJobs and leave the rest running:

for cj in $(kubectl get cronjobs -n team-reports -o name); do
  kubectl patch "$cj" -n team-reports -p '{"spec":{"suspend":true}}'
done

A suspended CronJob creates no runs, so your monitoring should alert on it like any other silence.

Spread the burst at the source

Thousands of tasks set for 01:00 on the first of the month is a choice somebody made. Add jitter, and review any change that schedules that many runs at once.

Keep the monitor off the shared platform

A checker that runs on the scheduler it checks goes quiet with it. A Forge app using a scheduled trigger to verify other scheduled work would have stopped at 01:00 too. Put the detector outside the failure domain.

Recovering the missed window

Afterwards, go through every job due between 01:00 and 03:50 and sort it: ran, ran late, or never ran. Dropped Bitbucket pipelines can be re-run through the API:

curl -fsS -X POST \
  -H "Authorization: Bearer $BITBUCKET_TOKEN" \
  -H "Content-Type: application/json" \
  "https://api.bitbucket.org/2.0/repositories/acme/billing/pipelines/" \
  -d '{"target":{"type":"pipeline_ref_target","ref_type":"branch","ref_name":"main","selector":{"type":"custom","pattern":"nightly-export"}}}'

That helps only if the job processes everything since its last success. A job hard-wired to "the last hour" will re-run the wrong hour.

Frequently asked questions about shared scheduler outages

What caused the Atlassian scheduling incident on 1 October 2026? One consumer of Atlassian's shared job scheduling platform scheduled an abnormally high volume of tasks for the same moment, 01:00 UTC. The surge exceeded the platform's scaling limits and degraded scheduled work across Jira, Confluence, Bitbucket Cloud and Forge until 07:25 UTC.

Did Bitbucket and Forge replay the scheduled runs they missed? No. Bitbucket scheduled pipelines due between 01:00 and about 03:50 UTC did not run, and Atlassian advised waiting for the next run or triggering one by hand. Forge scheduled triggers resumed around 04:00 UTC without replaying the missed invocations.

How can monitoring tell a late run from a dropped one? Use a check-in monitor with a grace period set per job. A late run checks in eventually and the alert clears by itself. A dropped run never checks in, so the alert stays open until the next successful run.

How do I stop one team from overloading a shared Kubernetes scheduler? Give each team its own namespace with object count quotas on Jobs and CronJobs, and keep a tested way to suspend every CronJob in one namespace. Both a quota rejection and a suspension mean a run does not happen, so monitor for missing runs as well.

Why should the scheduling monitor not run on the same platform? A monitor that runs on the scheduler it watches stalls in the same outage and says nothing. An external check-in monitor keeps counting while the platform is down and alerts when an expected run does not arrive.

Further reading


Conclusion: One tenant's burst on a shared scheduler left Atlassian customers with pipelines that never ran and Forge triggers that skipped three hours. None of it showed up as an error in anyone's own job. If you depend on a hosted scheduler, watch each job from outside it with a grace period you chose on purpose. If you run the shared scheduler, assume one of your tenants will try this on the first of some month and make sure you can stop them.

Sources: Atlassian Bitbucket status incident, 1 October 2026, Atlassian Confluence status incident, Kubernetes resource quotas, Kubernetes CronJob docs, Bitbucket Pipelines API.

Share

Related posts

A bright data center floor with rows of black server racks, blue cables looping from overhead trays, an orange traffic cone and an empty chair by a small desk
ReliabilityWhen the Platform Pauses, Missed Cron Runs Do Not Come Back
A code editor on a dark screen at an angle, with a project file tree on the left listing node_modules, pages, package.json and tsconfig.json, and TypeScript imports and a React component open in the main pane
ReliabilityNestJS @Cron Jobs That Stop for Good: The cron 3.x Timer Bug
A dark monitor filled edge to edge with dense, colour-highlighted minified JavaScript, shot at an angle so the right side fades into black
ReliabilityCloudflare Cron Triggers Broke for Schedules You Had Just Changed

Set up your first monitor.
It'll take 30 seconds.