What happened to build.opensuse.org on 5 November
The Open Build Service, which runs build.opensuse.org, was down for about 40 minutes on 5 November 2026. The post-mortem came out the same day, and the move that caused the outage is one most of us would have made.
A DDoS against the request page started on the evening of 3 November. Each hit enqueued a background job, BsRequestActionWebuiInfosJob, and the queue grew for a day and a half. To clear it faster, the team went from four workers to nine. The backend ran out of memory and crashed. Connections from the application server to the database then failed so often that MariaDB blocked the application host.
What ended it was deleting the jobs.
The timeline in UTC
Nov 3 16:00 DDoS against the request page begins
Nov 4 12:50 first customer report of failing pages
Nov 4 17:21 team alerted about a continuously growing job queue
Nov 5 11:10-11:13 workers scaled from 4 to 9
Nov 5 11:25 first automated alert of request processing failing
Nov 5 11:26 incident declared
Nov 5 12:05 all pending BsRequestActionWebuiInfosJob removed
Nov 5 12:06 workers back to 4
Nov 5 12:11 incident resolved
Almost 18 hours passed between the queue alert and the scale-up. The scale-up was 12 minutes old when the site fell over.
Why more workers made it worse
The post-mortem's lesson is worth copying into your runbook: scaling up a background job queue affects every host involved in processing the jobs, and the host running the workers is only one of them.
Workers are not where the work happens
A queue worker is mostly a loop that asks other systems to do things. Each of these jobs called into the OBS backend. Nine workers could send it more than twice as many calls as four. The worker host coped. The backend didn't.
The database is the shared ceiling
Once the backend crashed, connection attempts from the application server failed in bulk. The post-mortem doesn't name the setting, but the symptom and the fix (flushing the host cache) match MariaDB's max_connect_errors. After 100 failed handshakes in a row from one host, the default, MariaDB refuses that host until someone runs FLUSH HOSTS. So the whole web tier got locked out.
What a backlog does to scheduled jobs
The OBS wiki describes periodic jobs triggered by clockworkd and run by delayed_job workers, some hourly, some daily. Plenty of Rails apps work this way. A clock process drops a row into a queue on time, and a worker picks it up later.
A periodic job is a row in the queue
After the enqueue, the scheduler is done. If thousands of other rows sit ahead of it on the same workers, the nightly cleanup waits behind them. The clock process logs a perfect run. The cleanup hasn't started.
Queue depth is a late signal
OBS had a queue alert. It fired at 17:21 and nobody acted until morning, which is understandable: a long queue isn't necessarily a problem. Age is. A queue of 50,000 rows that drains in ten minutes is fine. A queue of 200 whose oldest job has waited six hours is stuck.
mysql app_production -e "
SELECT queue,
COUNT(*) AS ready,
TIMESTAMPDIFF(SECOND, MIN(run_at), NOW()) AS oldest_wait_s
FROM delayed_jobs
WHERE failed_at IS NULL
AND locked_at IS NULL
AND run_at <= NOW()
GROUP BY queue;"
+-----------+-------+---------------+
| queue | ready | oldest_wait_s |
+-----------+-------+---------------+
| default | 48211 | 61840 |
| quick | 12 | 4 |
| mailers | 3 | 9 |
+-----------+-------+---------------+
Alert on oldest_wait_s, with a threshold per queue.
Draining a backlog without a second outage
Decide whether the backlog is worth running
Jobs an attacker generated are worth nothing. OBS deleted them and the incident was over six minutes later. Before you add a worker, ask which jobs still matter and drop the rest:
mysql app_production -e "
DELETE FROM delayed_jobs
WHERE locked_at IS NULL
AND failed_at IS NULL
AND handler LIKE '%ReportPreviewJob%';"
Leave rows with a locked_at value alone; a worker is running them.
Scale in steps and watch downstream
If the work has to run, add one worker at a time and watch the database and backend:
systemctl start app-worker@5.service
sleep 300
mysql -e "SHOW GLOBAL STATUS LIKE 'Threads_connected';"
ssh backend-01 free -m
Connections and memory climbing faster than the queue shrinks means stop.
Give scheduled work its own workers
Don't let a flood of one job type share workers with the nightly billing run. delayed_job workers take a queue list:
RAILS_ENV=production bin/delayed_job --queue=scheduled start
Capping how much of one job type can pile up is the first action item in the OBS post-mortem. It's the same idea as the per-tenant limits after the Atlassian scheduler outage.
Check in from the end of the job
Queue age won't tell you whether last night's job finished. Put the check-in on the last line of the job's own code, after the work:
class NightlyDigestJob < ApplicationJob
queue_as :scheduled
def perform
DigestMailer.send_all
Net::HTTP.get(URI("https://www.cronguard.app/api/ping/nightly-digest"))
end
end
Make the grace period the time the job may wait in the queue plus its normal run time. When a flood holds it back longer, the check-in is missing and you get paged before a customer writes in.
Frequently asked questions about job backlogs and scaling workers
What caused the Open Build Service outage on 5 November 2026? A DDoS on the request page filled the background queue with BsRequestActionWebuiInfosJob entries. The team scaled the workers from four to nine to clear it, the extra load ran the backend out of memory, and failing database connections led MariaDB to block the application host.
Why can adding workers to a job queue cause an outage? Workers mostly call other systems, such as a backend service and the database. More workers means more load on those systems, and if they have no headroom they fail first. The OBS team concluded that scaling a queue affects every host involved in processing the jobs.
How does a job backlog delay scheduled jobs? In many apps a clock process only enqueues the periodic job, and a worker runs it later. If thousands of other jobs sit ahead of it on the same workers, the scheduled job waits even though the clock fired on time.
Should I alert on queue length or queue age? Queue age, measured as how long the oldest ready job has waited, per queue. A long queue that drains quickly is harmless, while a short queue whose oldest job has waited hours means work is stuck.
How do I know a queued scheduled job actually ran? Send a check-in from the last line of the job's own code, after the work is done. Set the grace period to the allowed queue wait plus the normal run time, so a missing check-in raises an alert when the job is held back too long.
Further reading
- One Tenant's 01:00 Burst Stalled Atlassian's Shared Job Scheduler
- Durable Job Queues Are Replacing Cron: New Failure Modes
- When a Scheduled Job Runs Long: Overlap, Skip, or Replace
Conclusion: The attack filled the queue, but it was the jump to nine workers that pushed that queue into a backend with no headroom and got the web tier locked out of MariaDB. Next time your queue balloons, delete what isn't worth running before you touch the worker count. And for the jobs you schedule, measure how long the oldest one has waited and have each one check in when it is done.
Sources: OBS post-mortem, 5 November 2026, OBS wiki on delayed jobs, delayed_job README, MariaDB error 1129.