Background Jobs, Task Queues, and Scheduling
Why slow or unreliable work gets moved out of the request cycle, and how job queues, retries, and idempotent design make that safe.
Background Jobs, Task Queues, and Scheduling
The player loads from youtube-nocookie.com when you press play. Nothing reaches YouTube before that.
Not started
What you'll learn
- A request handler should only do work the user is actually waiting on; anything slower than that belongs in a background job so the HTTP response doesn't sit there blocking on it.
- A job queue decouples the moment work is requested from the moment it runs, using a producer that enqueues the job, a broker that holds it, and a worker that executes it.
- Queues typically guarantee at-least-once delivery, not exactly-once, which means any job you write can run twice — so the job's logic has to be idempotent, not just retried.
- Retries need exponential backoff and a cap, or a struggling downstream dependency gets hammered harder the more it's already failing.
- Cron-style scheduling and a job queue solve different problems: one decides *when* work starts, the other decides *how* it runs reliably once started — many systems need both together.
The request cycle has a deadline; some work doesn’t
A web request has an implicit contract: a client sends it, and expects a response back in a reasonable amount of time — a second, maybe a few, before a browser spinner starts to feel broken or a mobile client’s socket just times out. Most of what a backend does fits inside that window. Looking up a user, validating a payload, writing a row to the database — all fast.
But plenty of real work doesn’t fit. Resizing an uploaded image, generating a PDF report, calling a third-party API that sometimes takes 8 seconds to respond, sending a batch of emails — none of that is something the user is actually watching happen in real time. If your POST /signup handler creates the user’s account and generates a welcome email and waits for the email provider’s API to confirm it sent, you’ve tied the fate of a fast, reliable operation (creating a user) to a slow, less reliable one (an outbound HTTP call to someone else’s mail service). If that third-party API is slow today, your signup endpoint is slow today, even though nothing about creating a user actually changed.
The fix is to stop doing the slow part inline. Create the user, respond to the client immediately, and hand “send the welcome email” off to something that will do it shortly afterward, outside the request-response cycle. That’s a background job.
The building block: a queue, a broker, a worker
The simplest version of this is three things working together:
- A producer — your application code, running inside a request handler — decides a job needs to happen and pushes a small description of it (a job type plus whatever data it needs) onto a queue.
- A broker holds that job until something is ready to process it. This is usually a dedicated piece of infrastructure: Redis, RabbitMQ, or a managed queue like AWS SQS. Popular job-queue libraries (Sidekiq for Ruby, Celery or RQ for Python, BullMQ for Node) are built on top of a broker like this and add the job-specific bookkeeping — retries, scheduling, dead-letter handling — on top.
- A worker is a separate running process (or pool of processes) that pulls jobs off the queue and actually executes them. It isn’t part of your web server’s request-handling path at all; it can live on a different machine, scale independently, and keep running even if your web servers are busy or restarting.
This separation is the entire point. The producer’s job is just “describe the work and hand it off” — that’s fast and can’t fail the way the actual work can. The worker does the slow, failure-prone part, on its own schedule, with its own retry logic, far away from any user who’s waiting on an HTTP response.
A job can run twice — design for it
Here’s the detail that trips people up: most job queues promise at-least-once delivery, not exactly-once. Think about why. A worker picks up a job, does the work, and then tells the broker “done, you can discard this.” If the worker crashes after finishing the real work but before sending that acknowledgment — power failure, container killed mid-request, network blip — the broker never heard “done.” From the broker’s point of view, the job might have failed, so it reassigns it to another worker. That worker runs it again.
This means the job you write has to produce the same correct result whether it executes once or three times. That property is called idempotency, and it’s not a nice-to-have — it’s the actual contract a job needs to honor, the same way an API endpoint that creates a resource often needs to handle a duplicate request safely.
Concretely, this usually means keying the side effect on something stable and checking before you act:
- Sending an email: before sending, check (or have the email provider check, using a client-supplied idempotency key) whether an email with this exact job ID has already gone out. If yes, skip it.
- Charging a payment: don’t say “charge $20” — say “charge $20 for order #4821, using idempotency key
order-4821-charge.” A well-designed payment API (Stripe’s is a common example) will recognize the repeated key and return the original result instead of charging twice. - Writing a database row: use an
UPSERTkeyed on a natural identifier from the job, not a blindINSERT, so a repeat run updates the existing row instead of creating a duplicate.
A job that just does “send welcome email” with no deduplication key is a bug waiting for a crash to trigger it — it works fine in testing, where nothing ever crashes mid-job, and then duplicates a charge or an email in production the first time a worker gets killed at the wrong instant.
Retries, backoff, and giving up gracefully
Jobs fail for reasons that have nothing to do with your code: a downstream API times out, a network blip drops a connection, a third-party service is having a bad day. The fix is almost never “don’t retry” — it’s “retry correctly.”
Correctly means two things. First, exponential backoff: wait a short time before the first retry, longer before the second, longer still before the third, instead of retrying immediately and repeatedly. Immediate unlimited retries are actively harmful when the failure is because a downstream dependency is already struggling — hammering it harder with retries is how you turn a brief blip into a full outage. Second, a maximum attempt count: after some number of failed retries, stop. Move the job to a dead-letter queue — a holding area for jobs that couldn’t complete — where a human (or an alert) can look at it, rather than letting it retry forever and burn resources on something that’s never going to succeed on its own.
Scheduling: cron jobs are a different axis
Everything above is about reliably running a job once it’s been triggered. Scheduling is a separate concern: deciding when a job gets triggered in the first place, on a recurring basis, without a user action kicking it off. A nightly report generation, a weekly digest email, a cleanup job that purges expired sessions every hour — these aren’t triggered by a request, they’re triggered by the clock.
The traditional building block here is cron — a time-based scheduler, either the Unix daemon itself or a cron-syntax scheduler built into your job-queue library (most of them support this directly: “run this job type every day at 2am”). The two concerns compose: a cron schedule decides when to enqueue the job, and the job queue’s retry and idempotency machinery takes over from there to make sure it actually completes. If your nightly report job fails halfway through, you still want backoff and retries on that run, same as any other job.
Where this stops: one app, not many services
This lesson is scoped to deferring work within a single application — your own web server pushes a job, your own worker process picks it up. That’s a different problem from decoupling multiple independent services from each other, which is what message queues and async communication and pub/sub and event-driven architecture cover at a system-design scale — where the point isn’t just “run this later,” it’s “let service A publish an event without knowing or caring which services B, C, and D are listening.” The mechanics overlap (both use a broker, both deal with delivery guarantees and retries), but the problem being solved is different: one application doing its own slow work later, versus a whole architecture of services staying loosely coupled from each other.
Check yourself
Why move work into a background job instead of just doing it inline in the request handler?
Because the user is only waiting on the part of the response they actually need; anything slower (sending an email, resizing an image, generating a report) can happen after the response is already sent, which keeps the request fast and keeps one slow dependency from timing out the whole request.
What does it mean for a job queue to offer 'at-least-once' delivery, and why does that matter for how you write job code?
It means the system guarantees a job will be attempted at least once, but makes no promise it'll run exactly once — a worker can crash after finishing the real work but before telling the broker it's done, so the broker hands the job to another worker. Your job code has to produce the same correct end state whether it runs once or three times.
What makes a job idempotent, concretely?
Running it twice (or five times) with the same input produces the same end state as running it once — usually by keying the side effect (an inserted row, a sent email, a charged payment) on a unique identifier and checking first, so a duplicate attempt finds the work already done and skips it, rather than redoing it.
Why does a retry need exponential backoff and a maximum attempt count instead of just retrying immediately and forever?
Immediate, unlimited retries pile more load onto a dependency at exactly the moment it's already struggling — which is how a transient blip turns into an outage. Backoff spaces retries out with growing delays, and a cap sends a job that still can't succeed to a dead-letter queue for a human to look at, instead of retrying forever.
How is a job queue different from the message-queue and pub-sub patterns used for service-to-service communication?
A job queue defers work within a single application — the producer and the worker are typically the same codebase, just running in different processes. Message queues and pub-sub are about decoupling *separate services* from each other at a system-design scale, which is a related but distinct problem.
Go deeper
Notes
Notes are saved to your account.
Everything else on this page works without one — progress and bookmarks keep themselves in this browser and come with you when you sign in. Notes are the one thing worth not trusting to a browser you might clear.
Your notes could not be loaded.