Fault Tolerance, Retries, and Idempotency

By the end of this lesson, you will be able to...

• explain why retrying a side effect without stable identity can break a promise.

• trace how an idempotency key turns repeated attempts into one logical operation.

• design a bounded retry and repair policy for an uncertain workflow.

Idea in one sentence: Fault tolerance is not "try again forever"; it is retrying in a way that preserves the product promise.

Core Insight

Imagine booking a lab appointment.

There is one slot left at 10:00. You press Reserve. The page waits. Then it says:

We could not confirm your reservation.

You press Reserve again.

What should happen?

The naive answer is simple: if the first request failed, try again. But the previous lesson should make us suspicious. A timeout did not prove the reservation service failed. The service may have received the first request, created the hold, written it to the database, and sent a response that arrived too late.

If the second request looks like a brand-new operation, the system may reserve two slots, send two confirmations, or leave support with two records for one user intent.

The promise is small but strict:

one user click to reserve one slot
should create at most one reservation

Fault tolerance means keeping that promise when ordinary faults happen: timeouts, late responses, process restarts, duplicate messages, and workers that run more than once.

Retries help with temporary faults. Idempotency makes retries safe for side effects.

Plain meaning:

Repeating the same logical operation should not repeat the dangerous effect.

In this scenario:

Retrying the same reservation attempt should find or continue the first reservation attempt, not create a second hold.

Technical name:

This is idempotency. A side-effecting operation is idempotent when repeated attempts with the same identity produce one logical effect.

The Situation

Use this reservation workflow:

browser
  -> gateway
  -> reservation service
  -> reservations database
  -> email worker

The user wants one appointment:

user: user-42
slot: lab-10am
intent: reserve this slot once

The reservation service owns the official fact:

slot lab-10am is held by user-42
reservation id is r-88

The gateway owns a different fact:

the browser waited 2 seconds
the gateway did not receive a confirmation in time

Both facts can be true. The danger begins when the caller turns its local timeout into a new side effect.

The Naive Retry

The first version is easy to build:

browser -> gateway: reserve lab-10am
gateway -> reservation: create hold
gateway waits 2 seconds
gateway times out
browser shows "try again"

The user presses again:

browser -> gateway: reserve lab-10am
gateway -> reservation: create hold

If each request is treated as independent, the receiver sees two possible reservations:

request A:
  create reservation r-88

request B:
  create reservation r-89

Maybe the database has a strict unique constraint on the slot and prevents two holds for the exact same slot. Good. But the same retry pattern can still create duplicate emails, duplicate waitlist entries, duplicate payment authorizations, duplicate support tickets, or confusing "reservation failed" states next to a real hold.

The deeper bug is not only duplicate rows. The deeper bug is that the system lost the identity of the user's logical operation.

Check: After the first timeout, what is unsafe about sending a second reservation request with no relationship to the first one?

Think first, then reveal.

Answer: The reservation service may treat the second request as a new user intent. Retrying should repeat the same logical operation, not create a new one.

The Mechanism: Receiver Memory

The mechanism that helps is receiver-side memory keyed by stable operation identity.

The caller chooses an idempotency key:

idempotency_key: reserve:user-42:lab-10am:attempt-1
user_id: user-42
slot_id: lab-10am

The key must stay the same across retries of the same logical operation. It should not change just because the browser retried, the gateway retried, or a worker restarted.

The reservation service stores the key durably before or while it starts the side effect:

reservation_attempts

key                                  request_hash  status     reservation_id
reserve:user-42:lab-10am:attempt-1   h29c          running    null

When a request arrives, the receiver follows a small decision rule:

if key does not exist:
  atomically create attempt record
  start reservation work

if key exists and request_hash matches:
  if status is running: return pending
  if status is succeeded: return stored success
  if status is failed_final: return stored failure

if key exists and request_hash differs:
  reject idempotency conflict

The request hash protects against a subtle bug. If the caller reuses the same key but changes slot_id from lab-10am to lab-11am, that is not the same logical operation. The receiver should reject the mismatch instead of hiding a changed request.

The atomic create matters too. Two retry attempts can arrive at nearly the same time:

request A checks for key: not found
request B checks for key: not found
request A creates reservation
request B creates reservation

That is broken. The receiver needs one guarded step:

create key if absent
otherwise read existing key

In a database, this is often a unique constraint plus a transaction. In another store, it might be compare-and-set. The implementation varies. The promise is the same: only one request gets to be first for that key.

A Worked Trace

Now replay the same reservation with idempotency.

The retry did not create a second reservation. It recovered the answer for the same logical operation.

There is also an in-progress case:

retry arrives while status=running
receiver returns:
  reservation_pending

That is not failure. It is the receiver saying, "I recognize this operation, and I am not done with it yet."

So far, idempotency has changed the retry from "do this again" into "continue or report the same operation."

Check: Which part enforces the at-most-one reservation promise: the browser choosing a key, or the reservation service atomically creating or reading the key?

Think first, then reveal.

Answer: Both are needed, but the receiver-side atomic create-or-read enforces the promise. A key that is not durably checked by the side-effect owner is only decoration.

What This Changes

Before idempotency, the caller has a bad choice after a timeout. It can give up and maybe strand a real reservation. Or it can try again and maybe create a duplicate side effect.

After idempotency, the caller has a better question to ask:

What happened to this operation identity?

That question is much safer than:

Can you do this action again?

The important design decision is where the memory lives. If the browser remembers the key but the reservation service forgets it after a restart, the promise is weak. If the gateway remembers the timeout but the reservation service owns the reservation, the gateway still cannot decide final truth alone. The side-effect owner must keep durable memory of the operation.

This also changes how the user experience should behave. The page does not need to say "failed" just because the first attempt timed out. It can say "checking reservation" while the system asks the owner about the existing key. If the owner has succeeded, the user receives the same reservation. If the owner is still running, the user sees a pending state. If the owner has a final safe failure, the user can try a new operation.

That is the shape of fault tolerance here: keep progress possible, but keep the side effect attached to one identity.

Retry Policy

Idempotency makes retries safer. It does not make retries free.

If a dependency is slow and every caller immediately retries, traffic can multiply:

100 original requests
100 first retries
100 second retries
100 third retries

The retry storm can overload the same service the retries are trying to heal.

A retry policy should answer five questions:

what identity makes the retry safe?
which failures are retryable?
which failures are final?
how many attempts fit inside the deadline?
what state survives after live retries stop?

For the reservation workflow:

safe identity:
  idempotency key for one reservation attempt

retryable:
  timeout
  connection reset
  503 service unavailable
  429 rate limited, after waiting

not retryable unchanged:
  slot does not exist
  user is not allowed
  request hash conflicts with existing key
  slot already owned by another confirmed reservation

live retry budget:
  at most 3 attempts inside 5 seconds
  backoff with jitter between attempts

after live retries stop:
  show reservation_pending
  schedule owner check or reconciliation

Backoff means waiting longer between attempts. Jitter means adding small randomness so every caller does not retry at the same moment. Deadlines stop retries from living forever inside one user request.

Repair After The Request

Some faults outlive the browser request.

The user closes the tab. The reservation service restarts after creating the hold but before sending the response. The email worker sends a confirmation, crashes, and later receives the same job again.

The workflow needs durable states:

new
  -> reserving
  -> reserved
  -> confirmation_pending
  -> confirmed

reserving
  -> reservation_pending
  -> failed_safely

reservation_pending
  -> reserved
  -> failed_safely
  -> needs_review

reservation_pending is not a vague shrug. It is a repairable state. A later worker can ask the reservation owner:

does key reserve:user-42:lab-10am:attempt-1 exist?
what status does it have?
is there a reservation id?
was a confirmation email already sent?

The repair path should also be idempotent. If the reconciliation worker runs twice, the second run should observe the same state or do harmless work. Otherwise the repair path can create the duplicate effects that the live path avoided.

Trade-offs and Limits

The trade-off is that idempotency makes retries safer by adding durable coordination work to the side-effect owner.

Idempotency improves safety for repeated side-effecting operations.

It costs durable storage, retention policy, request matching, and careful receiver-side enforcement.

It can still fail when the receiver stores keys only in memory, expires keys too early, forgets to guard concurrent creates, or treats different payloads as the same operation.

It does not solve every coordination problem. Idempotency protects one logical operation from being repeated. It does not decide global ordering across many operations. It does not elect a leader. It does not make replicas agree on a shared log. Those problems lead into consensus and quorums in the next lesson.

You can see the boundary when duplicate key hits rise, pending attempts age beyond the deadline, idempotency conflicts spike, or retry traffic becomes a large fraction of normal traffic.

Common Confusions

Confusion: Retrying means doing the operation again

Why it is tempting:

That is how retry often feels from the caller side.

Better model:

For side effects, retry should mean "continue or recover the same logical operation."

Confusion: A trace id is an idempotency key

Why it is tempting:

Both are identifiers attached to requests.

Better model:

A trace id helps follow one attempt through logs. An idempotency key must remain stable across retries and must be enforced by the side-effect owner.

Confusion: Idempotency is only a client feature

Why it is tempting:

The client sends the key.

Better model:

The receiver enforces idempotency by storing and checking the key durably before repeating the side effect.

Confusion: Pending means the system failed

Why it is tempting:

Pending feels less satisfying than success or failure.

Better model:

Pending is often the honest state while the owner still has stronger evidence than the caller.

Practice

Pick one workflow with a visible side effect: reserving a seat, creating an account, charging a card, sending an invitation, or submitting an application.

Write:

user-visible promise:
side-effect owner:
idempotency key:
fields included in the request hash:
atomic guard:
retryable failures:
non-retryable failures:
live retry budget:
state after live retries stop:
repair evidence:
signal that retries are causing harm:

Then change one parameter: reduce the idempotency-key retention time from 24 hours to 5 minutes.

Predict what improves and what gets worse.

A good answer should mention that shorter retention saves storage, but can reopen the duplicate window for late retries, delayed callbacks, or human repair. It should also name the receiver-side guard, not only the key format.

Resources

• [ARTICLE] Timeouts, Retries, and Backoff with Jitter
  • Focus: How retries, backoff, jitter, timeouts, and overload interact in production systems.
• [DOC] Stripe API: Idempotent Requests
  • Focus: A concrete API design for making side-effecting requests safe to retry.
• [BOOK] Designing Data-Intensive Applications
  • Focus: Reliability, fault tolerance, and why distributed systems need explicit recovery paths.

Key Takeaways

• Fault tolerance preserves the product promise when ordinary faults interrupt the happy path.
• Retrying a side effect is safe only when the receiver recognizes the same logical operation.
• Idempotency needs stable keys, request matching, durable receiver memory, and atomic create-or-read behavior.
• Retry policy must be bounded with deadlines, backoff, jitter, and clear retryable versus non-retryable failures.
• When live retries cannot finish, pending states and repair loops keep the workflow honest.