Delivery Semantics and Retry Discipline

By the end of this lesson, you will be able to...

• Compare at-most-once, at-least-once, and effectively-once claims as application design boundaries.

• Trace how acknowledgments, retries, deduplication, and DLQs change user-visible side effects.

• Review a retry policy by naming what can be lost, duplicated, delayed, or safely skipped.

Idea in one sentence: Delivery semantics are not magic broker promises; they are explicit choices about loss, duplication, acknowledgment, retry, and side-effect safety.

Core Insight

An order service publishes:

OrderPaid(order_id=O-812, payment_id=P-91, event_id=evt-202)

The email service should send one receipt.

The support dashboard should show that the order was paid.

The analytics service should count the payment.

The naive review says:

Make sure the event is delivered exactly once.

That sounds like a clean requirement. It is usually the wrong first question.

The better review starts with the side effect:

What bad thing happens if this message is lost?
What bad thing happens if this message is delivered twice?
Who records enough state to make a retry safe?
When should we stop retrying and ask for human or operational review?

Delivery semantics are boundary claims. They say what this part of the system is allowed to lose, duplicate, retry, or quarantine. A broker can provide useful guarantees, but the application still owns the user-visible consequence.

For this lesson, use one rule:

At-least-once delivery requires duplicate-safe consumers.
Retry discipline decides which duplicates are acceptable and which failures need a different path.

What You Can Now See

The first seven lessons gave us the pieces:

This review combines them into one delivery question:

What does the system promise when a message moves, fails, and moves again?

Plain meaning:

Delivery semantics describe what the sender, broker, and consumer can claim about a message reaching its destination.

In this scenario:

The broker may redeliver evt-202 if the email service crashes before acknowledging it.

Technical name:

That is an at-least-once delivery shape. The message should not disappear silently, but the consumer may see it more than once.

The Concepts Together

Here is the compact map.

At-most-once is useful when duplicates are worse than loss.

Example: a low-value telemetry point may be dropped rather than retried forever.

At-least-once is useful when loss is worse than duplicates.

Example: a payment fact should keep moving until a consumer handles it or an operator can inspect it.

Effectively-once is not the same as "the whole distributed system executed exactly once."

It means the application made one effect safe:

same event_id -> same protected side effect -> no duplicate user harm

For the receipt email, that means the email service records that evt-202 already produced receipt_sent.

Synthesis Example

Start with this delivery path:

Order service -> orders log -> email service -> email provider

The email service has a local table:

processed_events(event_id, side_effect)

Now trace one failure.

Output:

The user receives one receipt.
The broker may have delivered twice.
The application effect happened once.

Naive failure contrast:

If the email service sends first and records dedup state later, a crash between those actions can still create a duplicate email. If it records first and then fails before sending, it may skip a receipt that never went out.

That is why application-level delivery design needs a boundary. Some side effects need a transactional outbox, provider idempotency key, or reconciliation job. Those deeper implementation details belong in later tracks, but this track must make the risk visible.

So far:

Delivery retry handles uncertain movement.
Idempotency handles repeated observation.
Operational review handles messages that cannot be made safe by more retries.

Common Confusions

Confusion: "At-least-once means the consumer succeeded"

Why it is tempting:

The message eventually reached a consumer, so it feels complete.

Better model:

At-least-once is about delivery attempts and acknowledgments. It does not prove that every downstream side effect succeeded. The consumer may need its own durable state, idempotency key, or repair path.

Confusion: "Retries are always safer than dropping"

Why it is tempting:

Retries look like persistence. If something failed, trying again feels responsible.

Better model:

Retries help transient failures. They can make permanent failures louder and can multiply duplicate side effects. A retry policy needs a stop condition, a backoff rule, and a destination for messages that need review.

Confusion: "A DLQ fixes bad events"

Why it is tempting:

Moving a poison event into a dead-letter queue clears the main consumer path.

Better model:

A DLQ is a quarantine and investigation boundary. It preserves evidence and protects the hot path, but someone still needs to inspect, repair, replay, skip, or compensate.

Retrieval Check

Check: A consumer receives OrderPaid twice because it crashed before acknowledging the first delivery. Which claim did the broker probably provide: at-most-once or at-least-once?

Think first, then reveal.

Answer: At-least-once. The broker avoided silent loss by redelivering after an uncertain acknowledgment. The application must make the consumer side effect duplicate-safe.

Check: A retry policy keeps retrying a permanently invalid event every second. Lag grows and no other events for that partition move. What review decision is missing?

Think first, then reveal.

Answer: The policy is missing a stop condition and a quarantine path, such as a DLQ with evidence for review. More retries do not fix a permanent validation failure.

Trade-offs and Limits

Delivery discipline improves reliability by turning hidden uncertainty into explicit choices.

It costs storage, code, and operational attention.

This helps when:

• losing the message would violate a user or business promise
• duplicate side effects can be recognized by a stable event or command key
• operators need evidence when automatic retry is no longer safe

It costs:

• deduplication tables or idempotency keys
• retry budgets, backoff rules, and monitoring
• cleanup policies for old processed-message records
• careful design around non-transactional side effects

It does not solve:

• wrong event boundaries
• missing ownership
• invalid schemas
• side effects that cannot be checked or reconciled
• broker internals such as replication, transactional producers, or stream-processing exactly-once implementation

Signals to watch:

The trade-off is not "stronger is always better."

The trade-off is:

Less loss usually means more duplicate handling.
Less duplication usually means more state and stricter boundaries.
Less operational noise usually means clearer retry budgets and quarantine rules.

Transfer Challenge

Review this policy:

Event: PaymentCaptured(order_id, payment_id, event_id)
Consumer: fulfillment service
Side effect: reserve stock and create shipment request
Policy: retry every failure forever until the consumer returns success
Dedup state: none
DLQ: none

Write a better review note.

A good answer should mention:

• whether loss or duplication is more dangerous
• which side effect needs a stable deduplication key
• which failures should be retried with backoff
• which failures should stop automatic retry and move to review
• what signal would show the policy is hurting the system

Model answer:

This should not retry forever without deduplication. Losing PaymentCaptured may leave a paid order unfulfilled, so at-least-once delivery is reasonable. But reserving stock and creating shipment requests are user-visible side effects, so the consumer needs a stable key such as payment_id or event_id for deduplication. Transient database or provider failures can retry with backoff and a retry budget. Permanent validation failures, impossible inventory states, or repeated ambiguous provider results should move to a DLQ or operational review path. Watch duplicate shipment requests, stock reservations for the same payment, rising retry counts, and lag caused by one poison event.

What Comes Next

This review closes the first half of the track.

You can now ask a useful delivery question before adding more process design:

If this event moves twice, arrives late, or cannot be processed, who owns the consequence?

The next lesson uses that question to compare choreography and orchestration. Once many services react to events, delivery safety is not enough. The system also needs visibility into the business process.

Resources

• [BOOK] Martin Kleppmann - Designing Data-Intensive Applications
  • Link: https://dataintensive.net/
  • Focus: Use the messaging and stream-processing chapters to connect delivery guarantees with duplicate handling and recovery.
• [DOC] Apache Kafka Documentation - Message Delivery Semantics
  • Link: https://kafka.apache.org/documentation/#semantics
  • Focus: Read the semantics as concrete examples of producer, broker, and consumer boundary claims.
• [ARTICLE] Chris Richardson - Idempotent Consumer Pattern
  • Link: https://microservices.io/patterns/communication-style/idempotent-consumer.html
  • Focus: Connect duplicate message handling to stored processed-message records.
• [REFERENCE] Enterprise Integration Patterns - Dead Letter Channel
  • Link: https://www.enterpriseintegrationpatterns.com/patterns/messaging/DeadLetterChannel.html
  • Focus: Treat DLQs as explicit failure-routing and investigation boundaries.

Key Takeaways

• Delivery semantics describe loss, duplication, acknowledgment, and retry boundaries; they do not erase application responsibility.
• At-least-once delivery is common because loss is often worse than duplication, but it requires duplicate-safe consumers.
• Effectively-once is a local application effect, not proof that an entire distributed workflow ran exactly once.
• Retry discipline needs classification: transient failures, permanent failures, and ambiguous completion should not all get the same loop.
• A DLQ protects the main path and preserves evidence, but it is a review boundary, not a repair by itself.