Worker Retry Semantics, Dead Letters, and Poison Jobs
LESSON
Worker Retry Semantics, Dead Letters, and Poison Jobs
By the end of this lesson, you will be able to...
Trace an asynchronous job from delivery through a failed attempt, retry, durable effect, and acknowledgement.
Separate a safe retry from an ambiguous side effect and define the idempotency evidence each needs.
Choose a retry budget, dead-letter route, and operator signals for work that cannot make progress.
Idea in one sentence: A worker retry is safe only when the system can tell whether the intended effect already happened and can stop a permanently failing job from circulating forever.
Core Insight
Atlas Shop places charge-order:O-194 on a queue after checkout. A worker receives it, asks the payment provider to charge €49, and the provider completes the charge. Then the worker loses its network response and crashes before it acknowledges the queue message.
Later, the queue delivers the same job again. The simple rule “a failure means retry” is tempting because it preserves work after a crash. It is correct for a request that definitely did not reach its dependency. Here it can create a second €49 charge: the worker cannot tell whether the first request failed before the provider acted or succeeded just before the response was lost.
The stronger model is not “exactly once because there is a queue.” It is a lifecycle with explicit delivery semantics, an idempotent business effect, bounded retries, and a quarantine path for work that keeps failing. The queue can make a message visible again. The application must make repeating its effect safe or discoverable.
The Production Symptom
At 10:02, support sees a customer report two charges for order O-194. The worker dashboard looks healthy: the queue drained, worker CPU is low, and the job has a successful log line. Those facts do not prove that the charge happened only once.
The useful investigation questions are smaller:
- Which durable job identity represents this order's payment intent?
- Did the provider receive the first attempt, and under which idempotency key?
- Did the worker record a durable outcome before acknowledging the message?
- How many deliveries occurred, and did two workers overlap after a visibility timeout?
Delivery and completion are different events. A queue can say “a worker may process this message.” It cannot, by itself, prove what an external payment system did. That distinction is the reason retries, visibility timeouts, and dead letters need an operational design rather than just a configured number.
What the User Sees
The customer needs one clear result: either one charge with a receipt, or a visible pending state while the system resolves uncertainty. A silent duplicate is worse than a delayed confirmation.
For Atlas, the promise is therefore not “every job succeeds quickly.” It is: one order has at most one accepted charge intent, and any uncertain attempt becomes visible to support rather than being blindly repeated. This wording changes the worker's job. It must preserve the business identity payment:O-194, not merely remember a queue receipt that changes each delivery.
The Naive Retry Rule
The first worker loop is short:
receive message
call payment provider
delete message if the call returned success
otherwise let the message reappear
This fits a harmless task such as rebuilding a disposable search index. Repeating the work produces the same useful state, and a later attempt can replace an earlier one.
It breaks for a non-idempotent external effect. A timeout after call payment provider is ambiguous. The worker sees no success response, but the provider may have charged the card. Increasing the number of retries turns uncertainty into a larger duplicate-charge risk.
Another tempting fix is a long visibility timeout. A visibility timeout hides a received message from other workers for a period; if the worker does not delete it before expiry, it can become visible again. A timeout that is too short can create overlapping attempts; one that is too long delays recovery after a crash. It changes delivery timing, not whether the side effect is safe to repeat. Amazon SQS documents this at-least-once boundary explicitly: a message can be delivered more than once even during its visibility period. SQS visibility timeout.
The Failure Mechanism
Atlas gives every charge intent a durable idempotency key, charge:O-194. “Idempotent” here means that repeating the same intent produces the same accepted business outcome rather than a second charge. It is not the same as retrying a new order with a new key.
The payment provider must participate, or Atlas must have another authoritative way to query and reconcile the charge. A provider API can store the first result for an idempotency key and return that result on a later retry; Stripe documents this as a way to retry a request after a connection error without performing the operation twice. Stripe idempotent requests.
Atlas's illustrative lifecycle makes the intermediate states visible:
| Step | Queue / worker state | Durable payment state | What the next attempt may conclude |
|---|---|---|---|
| 1 | charge-order:O-194 is received; visibility starts. |
charge:O-194 is pending. |
One worker may attempt the provider call. |
| 2 | Worker sends provider request with charge:O-194; network response is lost. |
Provider may have recorded the charge, but the worker does not know yet. | This is an unknown outcome, not proof of failure. |
| 3 | Worker crashes; it cannot acknowledge the message. | The durable intent and key remain. | After visibility expires, another delivery is expected. |
| 4 | Retry receives the same job. | It uses the same idempotency key or reconciles provider state. | It obtains the existing charge outcome or creates it once. |
| 5 | Worker records succeeded durably, emits the receipt event, then acknowledges the queue message. |
The intent has a stable outcome. | A later duplicate delivery can see succeeded and do no new charge. |
The exact order of local writes and acknowledgement depends on the queue and database. The important invariant is visible: the acknowledgement is not the proof of payment; a durable, queryable business outcome is. If receipt delivery has its own external side effect, it needs its own idempotent identity or an outbox-like handoff. One idempotency key does not automatically protect every downstream action.
Investigation Path: Classify Before Retrying
Retries consume queue capacity and dependency capacity. Before choosing a delay, classify the failure from evidence:
| Evidence from an attempt | Working classification | Next action |
|---|---|---|
| Provider returns temporary overload or a connection failed before a request was sent. | Transient. | Retry with bounded backoff and jitter. |
| Provider returns “card declined” or the job payload lacks a required field. | Permanent for this intent. | Do not repeat automatically; mark failed or send to review. |
| Response is lost after the request may have reached the provider. | Unknown / ambiguous. | Reuse the idempotency key or reconcile the provider outcome before another effect. |
| The same malformed payload fails on each delivery. | Poison job. | Stop cycling it and quarantine it with evidence. |
The labels are a teaching model; a real integration needs provider-specific error semantics. What matters is that “timeout” does not always mean “safe to try again.” When the effect is expensive or irreversible, ambiguous outcomes deserve a reconciliation path, not a faster retry loop.
Mitigation and Prevention
Atlas configures an example retry schedule: immediate attempt, then waits of 30 seconds, 2 minutes, and 10 minutes, each with small randomized jitter. The delays are illustrative. Backoff gives an overloaded dependency time to recover, while jitter prevents a large group of failed jobs from retrying in lockstep.
It also sets a receive-attempt budget. After four unsuccessful deliveries, a message moves to a dead-letter queue (DLQ) with the job id, attempt count, error class, provider request id when known, and the code or configuration version that handled it. A DLQ is a quarantine and investigation queue, not a waste bin. In SQS, a redrive policy uses maxReceiveCount to move a repeatedly received source message to a DLQ; the documentation recommends allowing enough receives for legitimate recovery and then examining or redriving the moved work. SQS dead-letter queues.
For a poison job, an operator should answer three questions before redriving it: what caused the repeated failure, whether the earlier side effect happened, and what changed that makes a new attempt safe. Fixing a parser bug may make a redrive appropriate. Redriving an unknown payment charge without reconciliation simply moves the duplicate risk from the main queue to the DLQ.
Signals to Watch
Watch the whole lifecycle, not only queue depth:
| Signal | What it reveals |
|---|---|
| Delivery count and retry rate by error class | Whether transient failures are recovering or repeating. |
| Visibility-timeout expiries and concurrent attempts | Whether workers lose leases or processing exceeds its budget. |
| Age of the oldest ready job and delayed-retry backlog | Whether retry pressure is delaying new useful work. |
| DLQ arrival rate, age, and reason | Whether poison jobs or a systemic dependency failure need investigation. |
| Idempotency-key conflicts and reconciliation outcomes | Whether ambiguous effects are being resolved safely. |
| Provider latency and error rate alongside worker retries | Whether the worker is amplifying a downstream outage. |
The trade-off is persistence of effort versus duplicate side effects, queue pressure, and delayed recovery. More retries improve the chance that a transient failure completes. They can also keep bad work alive, amplify an outage, and hide a semantic failure until the queue becomes the incident. The useful boundary is not “five retries are always enough”; it is the point at which new evidence is unlikely to change the outcome without a human or a repaired dependency.
Readiness Check
Check: A worker calls an email provider, times out after the provider may have accepted the request, and the message becomes visible again. Is it safe to send the email immediately on the next delivery?
Think first, then reveal.
Answer: Not from the timeout alone. The second worker needs a stable email intent id and an idempotent provider operation, a query/reconciliation step, or a business decision that duplicate email is acceptable. The queue's repeated delivery only proves that the first worker did not acknowledge; it does not prove the provider did nothing.
Runbook prompt: A new deployment causes 2,000 jobs to enter the DLQ with the same validation error. First pause redrive for that job class so repaired workers do not create a retry storm. Inspect one representative payload, the deployed validation version, delivery counts, and any already-recorded side effect. Fix or roll back the validation issue, test one controlled replay with the same idempotency evidence, then redrive in a measured batch while watching DLQ arrivals and downstream latency.
Practice: Review a Refund Worker
A refund worker receives refund:R-88. The payment provider sometimes takes 90 seconds, the queue visibility timeout is 60 seconds, and the worker blindly retries every exception eight times. Design a safer lifecycle. State the stable identity, how you handle the 60-second boundary, which failures retry automatically, when work enters a DLQ, and what must be checked before a replay.
A good answer should mention:
- a durable refund intent such as
refund:R-88, reused as the provider idempotency key rather than a new key per delivery; - a visibility lease that is extended safely while known work is still running, or a timeout that fits the observed processing budget, while accepting that duplicates can still occur;
- bounded backoff with jitter for clearly transient provider failures, not immediate retries for every exception;
- reconciliation for an unknown result after a timeout, because a refund may already have occurred;
- a receive budget and DLQ record with error, attempt, provider reference, and code version; and
- a pre-redrive review of the cause, existing refund state, and downstream health.
Connections
The previous lesson limited how quickly traffic could enter a constrained path. This lesson limits how long failed asynchronous work may keep returning to that path. The capstone asks you to combine both boundaries with cache freshness, queue behavior, and evidence for tail latency.
Resources
- [DOCS] Amazon SQS visibility timeout — Focus: Trace receipt, temporary invisibility, expiry, duplicate delivery, and the timing trade-off.
- [DOCS] Amazon SQS dead-letter queues — Focus: Inspect receive budgets, quarantine, retention, investigation, and controlled redrive.
- [DOCS] Stripe idempotent requests — Focus: Separate retrying a stable business intent from creating a new external side effect.
Key Takeaways
- A redelivered job proves that acknowledgement did not happen; it does not prove that an external side effect did not happen.
- Use durable business identities and idempotent or reconcilable effects before retrying ambiguous outcomes.
- Visibility timeouts coordinate delivery timing, but they neither guarantee one delivery nor make an effect safe to repeat.
- Retry budgets, backoff, jitter, and DLQs preserve transient recovery while stopping poison jobs from consuming the main queue forever.
- A DLQ requires evidence and a redrive decision; replaying blindly can repeat the original incident.
← Back to Caching, Workers, and Performance