Background Queues and Job Processing
Design a job/worker architecture with retries, exponential backoff, dead-letter queues, delayed scheduling, and idempotent handlers so background work survives failures without corrupting data or looping forever.
Any work that doesn't need to finish before a response is sent back to the user — sending an email, resizing an uploaded image, generating a report, charging a card — belongs off the request/response cycle and in a background queue instead. A web request enqueues a small, serializable job description and returns immediately; one or more separate worker processes pull jobs off the queue and execute them independently, on their own schedule, with their own failure handling. Getting the happy path working (enqueue a job, a worker picks it up, it runs) is the easy 20%. The other 80% — what happens when a job fails, when it fails repeatedly, when it needs to run later rather than now, and when it accidentally runs twice — is what this lesson covers.
Job and worker architecture
The basic shape is a producer/consumer split: a producer (typically a web request handler) serializes a small payload describing the work to do — a job class name plus the data it needs, like ['job' => 'SendWelcomeEmail', 'userId' => 42] — and pushes it onto a queue (a database table, a Redis list, or a dedicated broker like RabbitMQ or Amazon SQS). One or more worker processes, running independently of any web request, continuously pull jobs off that queue and execute the corresponding handler. This separation is what keeps a web request fast (return in milliseconds, let the email actually send a second later) and lets background work scale independently of web traffic — add more workers when the job backlog grows, without touching the web tier at all.
The job payload should be the minimum needed to redo the work, not a snapshot of live objects — pass userId: 42, not a serialized User object, so the handler re-fetches current data when it actually runs (which might be minutes or hours after it was enqueued, by which point an embedded snapshot could be stale or the object's class could have changed shape).
• Producer enqueues a small, serializable payload; a separate worker process dequeues and executes it independently of any web request. • This keeps web requests fast and lets background capacity scale by adding workers, independent of web server scaling. • Pass IDs in the payload, not live object snapshots — the handler re-fetches current data when it actually runs. • Common backends: a database table polled by workers, a Redis list (LPUSH/BRPOP), or a dedicated broker like RabbitMQ or Amazon SQS.
Retry logic and exponential backoff
Background jobs fail for reasons that often have nothing to do with the job's own logic — a third-party API is briefly down, a database connection times out, a network blip drops a request mid-flight. These are transient failures, and the correct response isn't to give up immediately; it's to retry, since the same operation will very likely succeed a moment later once the transient condition clears. But retrying immediately, in a tight loop, is itself a problem: if the failure is because a downstream service is overloaded, an instant retry (multiplied across every failing job and every worker) adds more load to an already-struggling service, making the underlying problem worse.
Exponential backoff fixes this by waiting progressively longer between each retry attempt — for example 1 second, then 2, then 4, then 8, then 16 — so retry pressure on a struggling dependency decreases automatically over time instead of hammering it at a constant rate. Adding a small amount of random jitter to each wait (rather than a perfectly round number) also prevents many jobs that failed at the same instant from all retrying in lockstep, which would just recreate the load spike at each backoff step. Every retry policy needs a maximum attempt count — infinite retries turn a bad job into a job that runs forever.
• Retry transient failures (timeouts, temporary API/network errors) — they're likely to succeed on a later attempt. • Exponential backoff: each retry waits longer than the last (e.g. 1s, 2s, 4s, 8s...), easing pressure on a struggling dependency instead of hammering it. • Add jitter to backoff delays so many jobs that failed together don't all retry at the exact same moment. • Always cap the maximum number of attempts — a job that retries forever is a job that never actually fails, it just consumes resources indefinitely.
Dead-letter queues
Once a job exhausts its maximum retry attempts, it needs somewhere to go other than silently disappearing or looping forever — a dead-letter queue (DLQ) is exactly that: a separate holding area for jobs that failed every attempt, kept for inspection rather than discarded. Moving a job to a DLQ instead of dropping it preserves the full context of the failure (the payload, the error message from each attempt, the timestamps) so a person can look at it later, understand why it kept failing, and decide whether to fix the underlying bug and manually re-enqueue the job, or discard it as genuinely invalid.
A DLQ should trigger an alert, not sit silently — a job that failed five times and landed in the dead-letter queue is very likely evidence of a bug (a malformed payload, a downstream API that changed its response format) or an outage, and the whole point of the DLQ is to surface that to a human quickly rather than have the failure quietly disappear.
• DLQ = where jobs land after exhausting all retry attempts, instead of being silently dropped or retried forever. • Preserve the full failure context (payload, error messages, timestamps, attempt count) so someone can diagnose it later. • Alert on new dead-letter entries — they're a strong signal of a bug or an ongoing outage, not routine noise. • A human decides whether to fix-and-reprocess or discard a dead-lettered job; the system shouldn't guess.
Scheduling delayed and recurring jobs
Not every job should run the instant it's enqueued — sending a reminder email 24 hours after signup, retrying a failed job after a backoff delay, or running a nightly report all need to execute at a specific future time rather than immediately. The common implementation is to store an available_at (or run_at) timestamp alongside the job, and have workers only pick up jobs whose available_at has already passed; a Redis-backed queue can implement the same idea with a sorted set scored by the timestamp a job should become available, letting workers efficiently query only the jobs that are due.
Recurring jobs (run this report every night at 2am) are usually a thin layer on top of the same mechanism — a scheduler process wakes up periodically, checks which recurring definitions are due based on a cron-like expression, and enqueues a fresh one-off job for each, so the actual execution still goes through the same queue, retry, and dead-letter machinery as any other job.
• Delayed jobs: store an available_at/run_at timestamp; workers only dequeue jobs that are already due. • A Redis sorted set scored by due-time is a common way to implement an efficient delayed queue. • Recurring jobs: a scheduler enqueues a fresh one-off job when a cron-like schedule is due, reusing the same worker/retry/DLQ pipeline. • Backoff delays for a failing job are themselves just a special case of scheduling: re-enqueue with available_at set to now + backoff duration.
Idempotency: a job might run more than once
Most queue systems offer "at-least-once" delivery, not "exactly-once": a worker might successfully complete a job's side effects and then crash before it acknowledges/deletes the job from the queue, causing the queue to redeliver it to another worker even though it already ran. A job might also be manually re-run by a human recovering from a dead-letter entry, or retried by the backoff logic after a timeout that actually succeeded on the far end (the response was just lost on the way back). All of this means a well-designed job handler must be safe to run more than once with the same input and produce the same end state — this property is called idempotency.
The concrete technique is to record, durably, that a specific unit of work has already been done, and check that record before doing it again — typically via a unique idempotency key (an order ID, a payment intent ID) stored with a uniqueness constraint in the database, so a second attempt to process the same key either sees the existing record and skips the side effect, or the database itself rejects the duplicate insert. "Charge the customer's card" is the canonical example: a naive handler that just calls the payment gateway every time it runs will double-charge a customer the moment a retry happens after a network hiccup; an idempotent handler checks (atomically, via a unique constraint) whether this specific charge has already been recorded before calling the gateway at all.
• Most queues guarantee at-least-once delivery — a job can and eventually will run more than once for the same logical unit of work. • Idempotent = running the handler twice with the same input produces the same end state as running it once. • Technique: store a unique idempotency key with a database uniqueness constraint, and check/record it before performing the side effect. • Payment charges, sending a single email, and creating a resource are the classic operations that must be made idempotent — a duplicate is a real, user-visible bug, not a harmless no-op.
Code Examples
On failure, the job is re-enqueued with an incremented attempt count and a delay that doubles each time (plus a little jitter), rather than being retried instantly. Once attempt reaches MAX_ATTEMPTS, the job is written to a dead_letter_jobs table with its full payload and the error that kept occurring, instead of being retried forever or silently dropped — a person can query that table to see exactly what's failing and why.
$delaySeconds = (2 ** $attempt) + random_int(0, 1000) / 1000;
Common mistake: Retrying immediately in a loop with no backoff (or no max attempt count at all) turns one struggling downstream dependency into a self-inflicted denial-of-service against it, and a job with a genuine, permanent bug in its handler would retry forever, consuming worker capacity without ever making progress.
Before ever calling the payment gateway, the handler tries to insert a row keyed on idempotencyKey, protected by a database UNIQUE constraint. If this exact job has already been claimed — by an earlier attempt, or by another worker that raced to process the same delivery — the insert fails with a unique-constraint violation and the handler returns immediately without charging the card a second time.
VALUES (?, ?, ?, ?)' Common mistake: Calling $this->gateway->charge($customerId, $amountCents) directly, with no idempotency check, means that if the queue redelivers this job (worker crash after charging but before acknowledging, or a manual retry after a timeout that actually succeeded), the customer is charged again — a duplicate charge that's a real financial and trust problem, not a harmless side effect of 'at-least-once' delivery.
Jobs carry an available_at timestamp, and popDue() only selects rows whose available_at has already passed, so a job enqueued now with a 24-hour delay simply sits in the table, invisible to workers, until that time arrives. FOR UPDATE SKIP LOCKED lets multiple worker processes safely poll the same table concurrently without two workers claiming the same row. The same available_at mechanism is what powers the backoff retry example above — re-enqueuing a failed job with availableAt: time() + $delaySeconds is scheduling, just with a much shorter delay.
WHERE available_at <= ? AND locked_at IS NULL
<?php
declare(strict_types=1);
final class JobRunner
{
private const MAX_ATTEMPTS = 5;
public function __construct(
private readonly PDO $db,
private readonly QueueClient $queue,
) {}
public function process(array $job): void
{
try {
$this->dispatch($job);
} catch (Throwable $e) {
$attempt = $job['attempt'] ?? 1;
if ($attempt >= self::MAX_ATTEMPTS) {
$this->sendToDeadLetterQueue($job, $e);
return;
}
// Exponential backoff with jitter: base 2^attempt seconds,
// plus up to 1 extra second so retries don't cluster.
$delaySeconds = (2 ** $attempt) + random_int(0, 1000) / 1000;
$this->queue->push(
[...$job, 'attempt' => $attempt + 1],
availableAt: time() + (int) ceil($delaySeconds)
);
}
}
private function sendToDeadLetterQueue(array $job, Throwable $e): void
{
$this->db->prepare(
'INSERT INTO dead_letter_jobs (payload, error, failed_at) VALUES (?, ?, NOW())'
)->execute([json_encode($job), $e->getMessage()]);
…<?php
declare(strict_types=1);
final class ChargeCustomerJob
{
public function __construct(private readonly PDO $db, private readonly PaymentGateway $gateway) {}
/** $idempotencyKey is stable across retries — e.g. the order ID. */
public function handle(string $idempotencyKey, int $customerId, int $amountCents): void
{
// Atomically claim this idempotency key. The UNIQUE constraint on
// idempotency_key makes a duplicate INSERT fail instead of silently
// succeeding twice, even if two workers race on the same job.
try {
$this->db->prepare(
'INSERT INTO processed_payments (idempotency_key, customer_id, amount_cents, status)
VALUES (?, ?, ?, ?)'
)->execute([$idempotencyKey, $customerId, $amountCents, 'pending']);
} catch (PDOException $e) {
if ($this->isUniqueViolation($e)) {
// Already processed (or currently being processed) —
// safe to stop here. This is what makes a retry a no-op.
return;
}
throw $e;
}
$chargeId = $this->gateway->charge($customerId, $amountCents);
…<?php
declare(strict_types=1);
final class QueueClient
{
public function __construct(private readonly PDO $db) {}
public function push(array $payload, ?int $availableAt = null): void
{
$this->db->prepare(
'INSERT INTO jobs (payload, available_at, attempt) VALUES (?, ?, ?)'
)->execute([
json_encode($payload),
$availableAt ?? time(),
$payload['attempt'] ?? 1,
]);
}
/** A worker calls this in a loop to pull only jobs that are due. */
public function popDue(): ?array
{
$stmt = $this->db->prepare(
'SELECT id, payload FROM jobs
WHERE available_at <= ? AND locked_at IS NULL
ORDER BY available_at ASC LIMIT 1 FOR UPDATE SKIP LOCKED'
);
$stmt->execute([time()]);
$row = $stmt->fetch(PDO::FETCH_ASSOC);
return $row !== false ? json_decode($row['payload'], true) : null;
}
}
// Enqueue a reminder to run 24 hours from now instead of immediately.
$queueClient = new QueueClient($db);
$queueClient->push(
['type' => 'SendReminderEmail', 'userId' => 42],
availableAt: time() + 86_400
);Why this matters in production
Every one of the four ideas in this lesson exists because of a real incident pattern teams eventually hit: retries without backoff turning a brief third-party outage into a self-inflicted overload; a job that fails forever silently consuming worker capacity with no DLQ to surface it; a report that was supposed to run at 2am running immediately and every time a deploy restarts the worker, because there was no available_at gate; and, most visibly to customers, a payment retried after a timeout that had actually already succeeded, charging a card twice because the handler had no idempotency check. None of these are exotic edge cases — they're the default behavior of a naive queue implementation under completely ordinary failure conditions (a network blip, a slow database, a worker restart), which is exactly why production-grade queue systems (and libraries like Laravel's queues or Symfony Messenger) build retries, backoff, DLQs, and delayed scheduling in as first-class features rather than leaving them to be reinvented per project.
public function handle(int $customerId, int $amountCents): void { $this->gateway->charge($customerId, $amountCents); }
Check a durable, uniquely-constrained idempotency key (e.g. the order ID) before calling the gateway, and skip the charge entirely if that key has already been processed.
Queue delivery is typically at-least-once, so this exact job can run more than once — a worker crash after charging but before acknowledging the job, or a manual retry, will call charge() again with no idempotency check, double-charging the customer for the same order.
Why does exponential backoff use progressively longer delays between retries instead of retrying immediately every time? A. It's a way to guarantee a job never runs more than once B. Immediate, constant-rate retries add sustained load to a downstream dependency that may already be struggling, worsening the very problem causing the failures; progressively longer delays ease that pressure over time C. PHP's queue functions require a minimum 1-second delay between any two operations D. Exponential backoff makes the job's payload smaller
B. Immediate, constant-rate retries add sustained load to a downstream dependency that may already be struggling, worsening the very problem causing the failures; progressively longer delays ease that pressure over time — A transient failure (an overloaded API, a slow database) gets worse if every failing job retries at a constant, immediate rate; spacing retries out with exponentially growing delays reduces pressure on the struggling dependency, giving it room to recover.
What is the purpose of a dead-letter queue? A. To permanently delete jobs that fail, freeing up storage B. To hold jobs that have exhausted all retry attempts, preserving their context for human inspection instead of retrying forever or silently dropping them C. To speed up job processing by skipping validation D. To store successfully completed jobs for auditing
B. To hold jobs that have exhausted all retry attempts, preserving their context for human inspection instead of retrying forever or silently dropping them — A dead-letter queue is where jobs land once they've failed every allowed retry attempt, keeping the payload and error history available for a person to diagnose and decide whether to fix-and-reprocess or discard — the alternative of retrying forever or dropping silently is worse in both directions.
Why must a job handler that charges a customer's card be idempotent, given that most queues offer at-least-once delivery? A. Because PHP's queue functions automatically retry every job exactly once regardless of success B. Because at-least-once delivery means the same job can legitimately run more than once (e.g. a worker crash after charging but before acknowledging), so a non-idempotent handler would charge the customer again on redelivery C. Because payment gateways reject any request without an idempotency key D. Idempotency is only relevant for jobs that read data, not jobs that write data
B. Because at-least-once delivery means the same job can legitimately run more than once (e.g. a worker crash after charging but before acknowledging), so a non-idempotent handler would charge the customer again on redelivery — At-least-once delivery guarantees a job won't be silently lost, but as a trade-off it can be delivered and executed more than once for the same logical unit of work — a handler that doesn't check a durable idempotency key before performing an irreversible side effect like a charge will duplicate that side effect on redelivery.
A ResizeUploadedImage job currently calls $this->gateway->charge(...) — wait, it resizes an image and writes the result to disk with a fixed filename derived from the original upload's ID. Explain why this particular job is naturally idempotent without any extra idempotency-key logic, and then describe one plausible job (in a different domain) where writing to a fixed, deterministic location would NOT be enough to make it safe to re-run.
The image resize is naturally idempotent because writing the resized file to a fixed, deterministic path (derived from the same upload ID) means a second run simply overwrites the first run's output with an identical result — there's no accumulating side effect and no external system being told 'this happened' a second time. A payment/charge job, by contrast, is not naturally idempotent this way: even if you write a 'payment processed' row to a fixed location, the act of calling the external payment gateway is itself an irreversible side effect that happens outside your own storage — you must check your own idempotency record BEFORE calling the gateway, not just before writing your own final state, because the gateway itself has no idea this is a retry unless you tell it.
Think about what 'running the operation twice produces the same end state' requires: for the image resize, does a second run change anything observable compared to the first run's result?