Replay, Backfills, DLQs, and Poison Events
LESSON
Replay, Backfills, DLQs, and Poison Events
By the end of this lesson, you will be able to...
Decide when to replay events, backfill derived state, quarantine failures, or stop processing.
Trace how a poison event reaches a dead-letter queue and what evidence is needed before re-driving it.
Design a recovery plan that avoids repeating unsafe external side effects.
Idea in one sentence: Replay is not just "run the events again"; it is a controlled recovery action that must protect downstream state and side effects.
Core Insight
An order service publishes OrderPlaced.
Three consumers react:
inventory-service -> reserves stock
email-service -> sends a confirmation
analytics-service -> updates revenue projections
For two hours, the analytics consumer had a bug. It treated currency: "EUR" as if it were currency: "USD".
The event log still has the original events. That looks comforting. A teammate says:
No problem. Replay the last two hours.
That may be correct for analytics.
It would be dangerous for email. It may be dangerous for inventory. Replaying events can rebuild a derived view, but it can also repeat side effects that were only supposed to happen once.
The operational question is not "can the broker send old messages again?"
The better question is:
Which consumers can safely process these events again, and what must be quarantined, fixed, or backfilled instead?
Replay, backfills, dead-letter queues, and poison events are recovery tools. They are useful because event-driven systems keep evidence of what happened. They are risky because old evidence can meet new code, changed schemas, duplicate delivery, and side effects with real users.
The Naive Recovery Plan
The naive plan is:
If a consumer failed, replay the events.
If replay fails, put the messages in a DLQ.
After the bug is fixed, re-drive the DLQ.
This plan is tempting because it sounds mechanical. The log contains the past. The consumer has code. Run the code over the past.
That works when the consumer is rebuilding internal, replaceable state. For example, analytics can often rebuild a revenue projection from facts that already happened.
It breaks when processing the event causes an external effect:
- sending an email
- charging a card
- reserving stock
- calling a shipping carrier
- notifying a customer
Those effects are not erased when you rewind the consumer. The event log can replay OrderPlaced. It cannot unsend a duplicate email or unreserve stock without a separate compensating action.
Plain meaning:
Replay means processing old events again.
In this scenario:
The analytics consumer reads the OrderPlaced events from the two-hour incident window and recalculates its projection.
Technical name:
That is event replay. A backfill is a planned replay or recomputation used to fill missing or incorrect derived state. A dead-letter queue, or DLQ, is a quarantine for messages the normal consumer could not handle. A poison event is an event that repeatedly fails because the data, schema, or business meaning is incompatible with the consumer.
What the System Knows
Before choosing a recovery action, make the evidence visible.
For the incident above, the team knows:
| Evidence | What it says | Recovery meaning |
|---|---|---|
orders.events retains 7 days |
Old OrderPlaced events are available |
Replay is possible for the incident window |
| Analytics bug affected EUR orders | Wrong projection, not wrong source facts | Recompute analytics derived state |
| Email consumer was healthy | Confirmations were already sent | Do not replay email side effects |
Inventory consumer uses event_id inbox keys |
Duplicate deliveries can be skipped | Re-drive may be safe if the key is stable |
| 53 messages are in analytics DLQ | Some events still fail after retry | Triage before re-drive |
This table changes the recovery conversation.
The team is no longer asking for a broad "replay the topic" button. It is deciding which consumer group, time window, event type, and side effect boundary are safe to touch.
Check: Why is "replay the topic" a worse instruction than "replay OrderPlaced for analytics from 10:00 to 12:00 UTC after resetting the projection table"?
Think first, then reveal.
Answer: The second instruction names the consumer, event type, time window, and target state. It limits the blast radius. A broad topic replay may send old events to consumers that already completed unsafe side effects.
A Worked Recovery Trace
Start with the incident window:
10:00 bug deployed
10:03 OrderPlaced O-101 EUR
10:04 OrderPlaced O-102 USD
10:08 OrderPlaced O-103 EUR
12:00 bug fixed
The broken analytics consumer wrote this derived table:
| Order | Event fact | Broken derived state |
|---|---|---|
| O-101 | 50 EUR | counted as 50 USD |
| O-102 | 40 USD | counted as 40 USD |
| O-103 | 70 EUR | counted as 70 USD |
The source events are still true. The derived table is wrong.
A careful backfill plan looks like this:
| Step | Input | Transition | Intermediate state | Output or decision | Naive failure contrast |
|---|---|---|---|---|---|
| 1 | Incident report | Identify affected consumer and window | Analytics only, 10:00-12:00 UTC | Do not touch email or inventory | Naive replay sends old orders to every subscriber |
| 2 | Current projection table | Mark affected rows stale or rebuild into a new table | Old numbers remain visible until replacement is ready | Choose reset or shadow rebuild | Naive replay adds corrected values on top of wrong values |
| 3 | Event log | Read OrderPlaced from the window |
Events arrive in log order for the selected partition scope | Recompute projection with fixed code | Naive replay assumes all consumers are idempotent |
| 4 | DLQ entries | Inspect failing messages by reason | Separate malformed data, old schema, and code bug cases | Fix data/code or keep quarantined | Naive re-drive repeats the same failure loop |
| 5 | Reconciliation | Compare rebuilt totals with source-of-truth orders | Difference should shrink to expected zero or explained delta | Publish recovery note and monitor | Naive recovery stops when the command exits |
The important part is step 2.
If analytics increments counters when it handles an event, replaying without resetting or rebuilding can double count. The correct operation may be:
delete affected projection rows -> replay events -> compare totals
or:
build corrected projection in a shadow table -> compare -> swap readers
Both plans treat replay as a state transition, not as a magic undo button.
So far, the pattern is:
source facts stay stable
derived state may be rebuilt
external effects need idempotency or must be excluded
DLQ entries need diagnosis before re-drive
Poison Events and DLQ Triage
A DLQ is not a trash can.
It is a quarantine.
The message in the DLQ is evidence that normal processing could not continue safely. Re-driving it without changing anything usually creates noise, lag, and more failures.
Common DLQ reasons include:
| DLQ reason | Example | Better response |
|---|---|---|
| Temporary dependency failure | Tax service timed out | Retry later with limits and backoff |
| Consumer code bug | EUR parsed as USD | Fix code, then replay affected window |
| Schema mismatch | Old event has no field the new consumer requires | Add compatibility handling or transform during backfill |
| Bad business data | Order references a deleted account | Quarantine and decide a business rule |
| Non-idempotent side effect | Email sent before crash, ack not recorded | Check inbox/idempotency record before re-drive |
A poison event is usually not "bad" in a moral sense. It is an event the current consumer cannot process successfully. Sometimes the event is malformed. Sometimes the consumer is too strict. Sometimes the event is old but valid under a previous contract.
The triage question is:
Will this message succeed if we try again unchanged?
If the answer is no, re-drive is not recovery. It is repeating the incident.
Check: A DLQ contains 500 events that all fail with missing field: amount.currency. The events are six months old and were valid when published. Should the team re-drive them immediately after restarting the consumer?
Think first, then reveal.
Answer: No. The failure is not likely to disappear by retrying. The consumer needs compatibility handling, or the backfill needs a documented transform/default rule. Re-driving unchanged events will probably return them to the DLQ.
Designing a Safe Recovery Path
Use this small checklist before replay, backfill, or DLQ re-drive.
1. Name the target state
Do not start with the command.
Start with the state you want:
analytics projection for EUR orders from 10:00-12:00 UTC is recalculated with the fixed currency logic
That is more useful than:
replay orders
2. Separate source facts from derived state
Source facts are events like OrderPlaced or PaymentCaptured.
Derived state is a projection, cache, search index, report, or materialized view built from those facts.
Backfills are safest when they rebuild derived state from stable source facts. They are most dangerous when they repeat irreversible effects.
3. Check idempotency before side effects
If a consumer may call the outside world, look for a stable idempotency key:
event_id
order_id + effect_type
provider idempotency key
consumer inbox record
If the key is missing, replay may need to exclude that consumer or run in a dry-run mode.
4. Limit the replay window
Use the smallest time range, event type, partition, tenant, or consumer group that fits the incident.
Small windows make reconciliation possible. Large windows hide new mistakes inside old data.
5. Reconcile after the run
A recovery action should leave evidence:
- rows rebuilt
- DLQ count reduced
- failed reasons changed or disappeared
- business totals match the source of truth
- duplicate side-effect count stayed flat
The command finishing successfully is not enough. The system must show that the intended state changed.
Trade-offs and Limits
Replay improves recoverability. It gives the team a way to rebuild state after a bug, missed deployment, or temporary outage.
It costs operational discipline.
You need retention long enough to cover the recovery window. You need compatibility with old event shapes. You need consumers that can distinguish "handle this fact again" from "repeat the external effect again."
Backfills can also create load. A consumer that normally handles 100 events per second may fall behind if a backfill sends 10 million old events through the same path as live traffic. That connects directly to the next lesson on lag and backpressure.
Replay does not fix an incorrect source fact. If the original event says the wrong customer was charged, rebuilding projections will faithfully reproduce the wrong fact. You need correction events, compensation, or a business repair process.
DLQs do not solve failure by existing. They buy time and preserve evidence. They can still become a hidden pile of unhandled business decisions.
You can see the boundary when:
- DLQ size grows after every re-drive.
- Backfill changes totals but reconciliation still does not match the source of truth.
- Duplicate emails, reservations, or provider calls increase during replay.
- Old events fail because new consumers forgot old schemas.
- Live consumer lag rises while recovery work runs.
The trade-off is direct:
Keeping events replayable buys recovery options.
It also forces teams to design idempotency, compatibility, retention, and operational controls before the incident.
Common Confusions
Confusion: Replay means the system goes back in time
Why it is tempting:
The word "replay" sounds like rewinding a video.
Better model:
Replay sends old facts through current code. External effects and current schemas may have changed. You are not returning to the old system; you are applying old inputs to the system you have now.
Confusion: A DLQ is the final error handler
Why it is tempting:
Moving a message out of the main queue can make the live consumer healthy again.
Better model:
A DLQ is a pause point for diagnosis. The team still needs ownership, triage, repair rules, and evidence before re-drive.
Confusion: Backfills are only a data-platform concern
Why it is tempting:
Large backfills often happen in data pipelines.
Better model:
Application event systems also build derived state: projections, caches, search indexes, notification records, and read models. This track cares about the application-level decision: what can be recomputed safely, and what must not be repeated.
Practice
Review this recovery request:
The notification consumer crashed for 45 minutes.
During the outage, 12,000 InvoicePaid events accumulated.
Some events were later retried.
The consumer sends receipts through an email provider.
The provider supports idempotency keys, but the current consumer does not store them.
A developer suggests replaying the whole InvoicePaid topic from midnight.
Design a safer recovery plan.
A good answer should mention:
- Identify the affected consumer and exact outage window before choosing a replay range.
- Avoid replaying the whole topic to unrelated consumers.
- Add or verify an idempotency key before sending receipt emails again.
- Use an inbox or sent-receipt record to skip already completed email side effects.
- Re-drive only messages that are known to be unprocessed or safely deduplicated.
- Reconcile with provider records and DLQ counts after the run.
Resources
- [BOOK] Martin Kleppmann - Designing Data-Intensive Applications
- Link: https://dataintensive.net/
- Focus: Read the material on logs, derived data, and stream processing as background for replay and recomputation.
- [ARTICLE] Martin Fowler - What do you mean by Event-Driven?
- Link: https://martinfowler.com/articles/201701-event-driven.html
- Focus: Use the distinctions between event notification, event-carried state transfer, event sourcing, and CQRS to decide what replay is rebuilding.
- [DOC] Google Cloud Pub/Sub dead-letter topics
- Link: https://cloud.google.com/pubsub/docs/dead-letter-topics
- Focus: Look at DLQs as operational quarantine, not as automatic repair.
- [DOC] Amazon SQS dead-letter queues
- Link: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html
- Focus: Pay attention to redrive policy, retention, and why repeated failure needs diagnosis.
Key Takeaways
- Replay is a recovery action over old facts, not a guarantee that every side effect can be safely repeated.
- Backfills are safest when they rebuild derived state from stable source events and then reconcile the result.
- DLQs preserve failed messages for diagnosis; re-driving without fixing the cause usually repeats the failure.
- Poison events reveal a mismatch between event data, consumer assumptions, schema compatibility, or business rules.
- Safe recovery plans name the target state, limit the window, protect idempotency, and verify operational evidence after the run.
← Back to Event-Driven Architecture and Streaming Foundations