Dependency Failure and Graceful Degradation

By the end of this lesson, you will be able to...

• Classify dependencies by the part of the user promise they can break.

• Design a degraded mode that preserves the most important promise when a dependency fails.

• Review a dependency outage plan for fallback behavior, timeouts, retries, and honest user communication.

Idea in one sentence: Graceful degradation means choosing a smaller honest promise before a dependency failure turns into total failure.

Core Insight

The checkout service has learned several reliability habits.

It has a user promise. It has SLIs and SLOs. It pages on symptoms. It watches capacity, saturation, and safety margin.

Then the payment provider starts timing out.

At first, checkout is still "up." The API responds. The database is healthy. CPU is fine.

But every checkout request waits on the payment provider.

user clicks Pay
  -> checkout API validates cart
  -> checkout API reserves inventory
  -> checkout API calls payment provider
  -> payment provider times out
  -> checkout API waits, retries, waits again
  -> user sees spinner
  -> user clicks Pay again

One external dependency is failing, but the local service is now at risk too.

Workers stay occupied. Queues grow. Retries add load. Users lose a clear result. Duplicate-charge risk rises.

The naive design says:

If payment is down, checkout is down.

Sometimes that is true. You cannot honestly say an order is paid if payment is unknown.

But "we cannot complete the ideal path" does not always mean "we must fail everything." A better design asks:

What smaller promise can we keep safely?

For checkout, the smaller promise might be:

If payment confirmation is unavailable, give the user a clear pending state
within 2 minutes, do not charge twice, and reconcile later.

That is graceful degradation.

Plain meaning:

Graceful degradation means the service keeps a reduced, honest behavior when one part fails, instead of collapsing into confusing or unsafe behavior.

In this scenario:

Checkout stops promising "paid now" when the provider is unhealthy. It promises "safely pending, no duplicate charge, clear next state."

Technical name:

That reduced operating behavior is a degraded mode.

The Naive Dependency Model

A simple dependency diagram often looks like this:

checkout -> payment provider
checkout -> inventory service
checkout -> fraud service
checkout -> email service

The naive model treats each arrow as equally required.

If a dependency is used by checkout, checkout needs it.
If checkout needs it, failure should fail the request.
If failure fails the request, the only answer is to wait or retry.

That model creates two problems.

First, it hides different kinds of promises.

Payment authorization is not the same as sending a receipt email.

Inventory reservation is not the same as fraud enrichment.

A user-facing final state is not the same as an internal analytics event.

Second, it makes retries look like the only reliability tool.

Retries can help when a failure is brief and independent. They can hurt when the dependency is slow or overloaded. A retry that waits too long can consume local workers. A retry loop can turn one dependency failure into local saturation.

A better model classifies each dependency by the promise it protects.

The design question is not:

Can this dependency fail?

It can.

The design question is:

When it fails, what exact user promise do we still keep?

Strong, Weak, and Optional Dependencies

A useful reliability review separates dependencies into three groups.

Strong dependency

The service cannot keep the core promise without this dependency.

Example:

Checkout cannot say "paid" without a trustworthy payment result.

Strong does not always mean "block forever." It means the ideal response is impossible without the dependency.

Weak dependency

The service can keep a smaller promise without this dependency.

Example:

Checkout cannot finish payment confirmation, but it can create a pending order
with an idempotency key and reconcile later.

Weak dependencies are where graceful degradation usually lives.

Optional dependency

The service can keep the user promise without this dependency.

Example:

Email receipt can be sent later.
Analytics can be buffered.
Recommendations can be hidden.

Optional dependencies should almost never block the core path during a failure.

Check: The fraud service is timing out. Low-risk orders can be marked pending for later review, but high-risk orders must not proceed. Is the fraud service strong, weak, or optional?

Think first, then reveal.

Answer: It is weak for some traffic and strong for other traffic. The degraded mode can allow low-risk orders into pending review while rejecting or delaying high-risk orders. Dependency strength can depend on request type and risk.

A Worked Degradation Plan

Design the degraded mode for a payment-provider outage.

Start with the normal promise:

Normal checkout promise:
  Users receive a clear result within 2 minutes:
  paid, declined, or safely pending.

Correctness promise:
  Users are not charged twice for one checkout attempt.

Now state the failure:

Dependency failure:
  payment provider timeout rate is above 30%
  provider p95 latency is above 8 seconds
  clear-result SLI is beginning to burn

The naive path is:

try payment
  -> wait
  -> timeout
  -> retry
  -> wait
  -> retry
  -> user waits
  -> worker stays occupied
  -> user retries manually
  -> duplicate risk increases

That path is dangerous because it creates both latency and correctness risk.

Now design the degraded path.

Trace one request.

Input:

User U submits checkout attempt A with idempotency key K.

Transition:

Checkout validates the cart and writes an order attempt.
Checkout calls payment provider with key K.
Provider does not answer within the bounded timeout.

Intermediate state:

Order state:
  payment_pending

Stored evidence:
  user ID
  order ID
  idempotency key K
  provider attempt ID if known
  timeout reason
  next reconciliation time

Output:

User sees:
  "Payment is pending. Do not retry this checkout.
   We will update the order when confirmation arrives."

Later transition:

Reconciler asks provider about attempt K.
If provider accepted payment, order becomes paid.
If provider declined payment, order becomes declined.
If provider state is still unknown after a limit, order goes to support review.

Naive failure contrast:

Without the degraded mode, the request waits too long, retries too much,
occupies workers, and invites the user to retry manually.
That can break latency, availability, and correctness at the same time.

The degraded mode does not make the provider healthy. It protects the local promise from becoming dishonest or unsafe.

Circuit Breakers, Fallbacks, and Honest Responses

Three design tools often appear in degraded modes.

Timeout

A timeout says:

We will not wait forever.

Without a timeout, a slow dependency can consume workers until the local service saturates.

Circuit breaker

A circuit breaker says:

This dependency is failing enough that we should stop sending normal traffic
for a short period.

It protects the dependency from more load and protects the caller from waiting on a path that is probably unhealthy.

For checkout:

If provider timeouts exceed the threshold,
open the circuit for normal payment attempts.

While open:
  create safe pending states
  do not run repeated user-path retries
  allow controlled probe requests to test recovery

Fallback

A fallback says:

Here is the alternative behavior while the normal path is unavailable.

A fallback must be honest.

Bad fallback:

"Payment succeeded" when the payment state is unknown.

Good fallback:

"Payment is pending. We will confirm soon. Do not retry."

The fallback should preserve the most important part of the promise.

For checkout, correctness is more important than pretending everything is fast. A clear pending state is less convenient than immediate success, but it is safer than duplicate charges or ambiguous failure.

Check: Why is "show success and fix it later" a bad fallback for an unknown payment result?

Think first, then reveal.

Answer: It lies about state. If the payment later fails, the user saw a false success. If the user retries, the system may create duplicate risk. A degraded mode must be smaller than the normal promise, but still truthful.

Designing the Degraded Promise

A degraded mode needs a written promise.

Use this shape:

When <dependency or capability> is unhealthy,
the service will still <preserved promise>,
but it will not <normal capability that is unavailable>,
and users/operators will see <honest signal>.

For checkout:

When the payment provider is timing out,
checkout will still give users a clear pending state within 2 minutes
and protect idempotency for the checkout attempt,
but it will not promise immediate paid or declined status,
and users will see pending payment with a clear next step.

Now connect that promise to controls.

This is design work, not only operations work.

The degraded behavior must exist before the incident. During the incident, the team should be choosing and operating known modes, not inventing the state machine under pressure.

Trade-offs and Limits

Graceful degradation improves reliability because it avoids all-or-nothing behavior.

It can preserve a smaller but valuable promise while a dependency is slow, overloaded, or unavailable. It can also prevent retries from turning an external failure into local saturation.

It costs product clarity and engineering complexity.

You need extra states, user messages, reconciliation logic, idempotency, dashboards, and support flows. You also need to test the degraded path. An untested degraded mode is often just a hopeful diagram.

It does not solve every dependency failure.

Some dependencies are truly strong for some promises. If payment cannot be trusted, checkout cannot honestly say "paid." If inventory cannot be known for a scarce item, the service may need to stop selling that item.

You can see the boundary when:

the degraded queue grows without catching up
support cannot explain pending states to users
the fallback hides important risk
reconciliation creates duplicate or contradictory states
the dependency failure lasts longer than the business can tolerate

The trade-off is:

Failing closed protects correctness but may hurt availability.
Failing open preserves flow but may hurt correctness.
Graceful degradation chooses the smallest honest promise for the situation.

Common Confusions

Confusion: "Graceful degradation means users do not notice"

Why it is tempting:

The word "graceful" can sound like invisible.

Better model:

Users may notice a reduced experience. The goal is not invisibility. The goal is honest, bounded, safer behavior.

Confusion: "Fallback means fake the normal result"

Why it is tempting:

Teams want the product to feel smooth.

Better model:

A fallback must be truthful. It can return cached, partial, delayed, or pending information, but it should not claim certainty the system does not have.

Confusion: "Retries are the degraded mode"

Why it is tempting:

Retries are easy to add and sometimes fix transient failures.

Better model:

Retries are one control. A degraded mode also needs timeouts, limits, state, user communication, observability, and a recovery path.

Confusion: "Every dependency can be made optional"

Why it is tempting:

Optional dependencies make architecture feel safer.

Better model:

Some promises require specific dependencies. Reliability design is about naming which promises can shrink and which promises must stop rather than lie.

Practice

Review this dependency outage plan.

Dependency:
  fraud scoring service

Failure:
  timeout rate above 40%

Current plan:
  retry fraud scoring three times in the user request
  if all retries fail, approve the order
  send an alert to the fraud team later

Answer:

1. What promise is at risk?
2. What is unsafe about the current plan?
3. What degraded mode would be safer?
4. What signal should show that the degraded mode is reaching its limit?

Model answer:

1. The promise at risk is not only checkout speed. It is also safe ordering:
   the service should not approve high-risk orders without enough evidence.

2. The current plan waits too long in the user path and then fails open for
   every order. That may preserve short-term availability but can damage
   correctness, fraud risk, and business trust.

3. A safer degraded mode could approve only low-risk orders under stricter
   limits, move uncertain orders to manual review or pending review, reject
   clearly high-risk orders, and avoid repeated user-path retries.

4. Useful limit signals include pending-review queue age, percentage of orders
   entering degraded review, fraud-service timeout rate, and SLO burn for clear
   checkout result. If the pending-review queue grows too old, the degraded
   mode is no longer preserving the promise.

Resources

• [BOOK] Site Reliability Engineering: Handling Overload
  • Link: https://sre.google/sre-book/handling-overload/
  • Focus: Connect dependency pressure, load shedding, and protecting the caller from overload.
• [BOOK] Site Reliability Engineering: Addressing Cascading Failures
  • Link: https://sre.google/sre-book/addressing-cascading-failures/
  • Focus: Watch how retries and waiting can spread a failure across services.
• [ARTICLE] AWS Builders Library: Timeouts, retries, and backoff with jitter
  • Link: https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/
  • Focus: Use it to reason about bounded retries and why retry storms happen.
• [BOOK] Release It!: Design and Deploy Production-Ready Software
  • Link: https://pragprog.com/titles/mnee2/release-it-second-edition/
  • Focus: Read for stability patterns such as circuit breakers, timeouts, bulkheads, and integration failure.

Key Takeaways

• A dependency failure should trigger the question: "What smaller honest promise can we still keep?"
• Graceful degradation is not fake success; it is a reduced behavior with truthful state, bounded waiting, and a recovery path.
• Classify dependencies by promise impact: strong, weak, optional, and sometimes different by request type.
• Timeouts, circuit breakers, fallback states, idempotency, retry caps, and reconciliation work together; retries alone are not a degraded mode.
• A degraded mode needs its own signals, especially queue age, pending state age, duplicate-risk signals, and SLO burn.