Back to blog

Adding Workers to Drain a Job Backlog Took OBS Offline

On 5 November 2026 the Open Build Service went from four queue workers to nine to clear a backlog. The backend ran out of memory and MariaDB locked out the app.

CronGuard TeamCron Job Monitoring Experts
6 min read
A close-up of a row of black perforated rack panels with bright blue lights shining through the ventilation slots

What happened to build.opensuse.org on 5 November

The Open Build Service, which runs build.opensuse.org, was down for about 40 minutes on 5 November 2026. The post-mortem came out the same day, and the move that caused the outage is one most of us would have made.

A DDoS against the request page started on the evening of 3 November. Each hit enqueued a background job, BsRequestActionWebuiInfosJob, and the queue grew for a day and a half. To clear it faster, the team went from four workers to nine. The backend ran out of memory and crashed. Connections from the application server to the database then failed so often that MariaDB blocked the application host.

What ended it was deleting the jobs.

The timeline in UTC

Nov 3 16:00        DDoS against the request page begins
Nov 4 12:50        first customer report of failing pages
Nov 4 17:21        team alerted about a continuously growing job queue
Nov 5 11:10-11:13  workers scaled from 4 to 9
Nov 5 11:25        first automated alert of request processing failing
Nov 5 11:26        incident declared
Nov 5 12:05        all pending BsRequestActionWebuiInfosJob removed
Nov 5 12:06        workers back to 4
Nov 5 12:11        incident resolved

Almost 18 hours passed between the queue alert and the scale-up. The scale-up was 12 minutes old when the site fell over.

Why more workers made it worse

The post-mortem's lesson is worth copying into your runbook: scaling up a background job queue affects every host involved in processing the jobs, and the host running the workers is only one of them.

Workers are not where the work happens

A queue worker is mostly a loop that asks other systems to do things. Each of these jobs called into the OBS backend. Nine workers could send it more than twice as many calls as four. The worker host coped. The backend didn't.

The database is the shared ceiling

Once the backend crashed, connection attempts from the application server failed in bulk. The post-mortem doesn't name the setting, but the symptom and the fix (flushing the host cache) match MariaDB's max_connect_errors. After 100 failed handshakes in a row from one host, the default, MariaDB refuses that host until someone runs FLUSH HOSTS. So the whole web tier got locked out.

What a backlog does to scheduled jobs

The OBS wiki describes periodic jobs triggered by clockworkd and run by delayed_job workers, some hourly, some daily. Plenty of Rails apps work this way. A clock process drops a row into a queue on time, and a worker picks it up later.

A periodic job is a row in the queue

After the enqueue, the scheduler is done. If thousands of other rows sit ahead of it on the same workers, the nightly cleanup waits behind them. The clock process logs a perfect run. The cleanup hasn't started.

Queue depth is a late signal

OBS had a queue alert. It fired at 17:21 and nobody acted until morning, which is understandable: a long queue isn't necessarily a problem. Age is. A queue of 50,000 rows that drains in ten minutes is fine. A queue of 200 whose oldest job has waited six hours is stuck.

mysql app_production -e "
SELECT queue,
       COUNT(*) AS ready,
       TIMESTAMPDIFF(SECOND, MIN(run_at), NOW()) AS oldest_wait_s
FROM delayed_jobs
WHERE failed_at IS NULL
  AND locked_at IS NULL
  AND run_at <= NOW()
GROUP BY queue;"
+-----------+-------+---------------+
| queue     | ready | oldest_wait_s |
+-----------+-------+---------------+
| default   | 48211 |         61840 |
| quick     |    12 |             4 |
| mailers   |     3 |             9 |
+-----------+-------+---------------+

Alert on oldest_wait_s, with a threshold per queue.

Draining a backlog without a second outage

Decide whether the backlog is worth running

Jobs an attacker generated are worth nothing. OBS deleted them and the incident was over six minutes later. Before you add a worker, ask which jobs still matter and drop the rest:

mysql app_production -e "
DELETE FROM delayed_jobs
WHERE locked_at IS NULL
  AND failed_at IS NULL
  AND handler LIKE '%ReportPreviewJob%';"

Leave rows with a locked_at value alone; a worker is running them.

Scale in steps and watch downstream

If the work has to run, add one worker at a time and watch the database and backend:

systemctl start app-worker@5.service
sleep 300
mysql -e "SHOW GLOBAL STATUS LIKE 'Threads_connected';"
ssh backend-01 free -m

Connections and memory climbing faster than the queue shrinks means stop.

Give scheduled work its own workers

Don't let a flood of one job type share workers with the nightly billing run. delayed_job workers take a queue list:

RAILS_ENV=production bin/delayed_job --queue=scheduled start

Capping how much of one job type can pile up is the first action item in the OBS post-mortem. It's the same idea as the per-tenant limits after the Atlassian scheduler outage.

Check in from the end of the job

Queue age won't tell you whether last night's job finished. Put the check-in on the last line of the job's own code, after the work:

class NightlyDigestJob < ApplicationJob
  queue_as :scheduled

  def perform
    DigestMailer.send_all
    Net::HTTP.get(URI("https://www.cronguard.app/api/ping/nightly-digest"))
  end
end

Make the grace period the time the job may wait in the queue plus its normal run time. When a flood holds it back longer, the check-in is missing and you get paged before a customer writes in.

Frequently asked questions about job backlogs and scaling workers

What caused the Open Build Service outage on 5 November 2026? A DDoS on the request page filled the background queue with BsRequestActionWebuiInfosJob entries. The team scaled the workers from four to nine to clear it, the extra load ran the backend out of memory, and failing database connections led MariaDB to block the application host.

Why can adding workers to a job queue cause an outage? Workers mostly call other systems, such as a backend service and the database. More workers means more load on those systems, and if they have no headroom they fail first. The OBS team concluded that scaling a queue affects every host involved in processing the jobs.

How does a job backlog delay scheduled jobs? In many apps a clock process only enqueues the periodic job, and a worker runs it later. If thousands of other jobs sit ahead of it on the same workers, the scheduled job waits even though the clock fired on time.

Should I alert on queue length or queue age? Queue age, measured as how long the oldest ready job has waited, per queue. A long queue that drains quickly is harmless, while a short queue whose oldest job has waited hours means work is stuck.

How do I know a queued scheduled job actually ran? Send a check-in from the last line of the job's own code, after the work is done. Set the grace period to the allowed queue wait plus the normal run time, so a missing check-in raises an alert when the job is held back too long.

Further reading


Conclusion: The attack filled the queue, but it was the jump to nine workers that pushed that queue into a backend with no headroom and got the web tier locked out of MariaDB. Next time your queue balloons, delete what isn't worth running before you touch the worker count. And for the jobs you schedule, measure how long the oldest one has waited and have each one check in when it is done.

Sources: OBS post-mortem, 5 November 2026, OBS wiki on delayed jobs, delayed_job README, MariaDB error 1129.

Share

Related posts

A black-and-white photo of a laptop at the right edge of a desk, its screen filled with a terminal system monitor showing memory bars, network graphs and a process list, next to a blurred bright window
ReliabilityOne Tenant's 01:00 Burst Stalled Atlassian's Shared Job Scheduler
A dark computer monitor showing blurred lines of code in orange and white, with a blue glow in the lower left corner and a red light strip beside a second screen on the right
ReliabilityA Kubernetes Job Recreated Under the Same Name Can Sit at Zero Pods
A bright data center floor with rows of black server racks, blue cables looping from overhead trays, an orange traffic cone and an empty chair by a small desk
ReliabilityWhen the Platform Pauses, Missed Cron Runs Do Not Come Back

Set up your first monitor.
It'll take 30 seconds.