Backpressure, Load, and Cascading Failure

By the end of this lesson, you will be able to...

• trace how one slow dependency can become overload across callers, queues, workers, and retries.

• choose where to use admission control, bounded queues, concurrency limits, deadlines, and load shedding.

• interpret operational signals that show whether a system is absorbing a burst or building a cascade.

Idea in one sentence: Backpressure protects a system by making overload visible and bounded before it turns accepted work into promises the system cannot keep.

Core Insight

A ride-hailing app shows drivers near a rider.

Every few seconds, the app asks an ETA service:

How long until driver D can reach rider R?

The ETA service calls a routing provider. Normally the provider answers in 80 ms. During a city event, the provider starts answering in 2 seconds. The ETA service does the friendly thing: it accepts every request and waits.

At first, this looks reasonable. Users want ETAs. Dropping requests feels bad.

Then the shape changes.

Clients time out and retry. Workers sit on slow calls. Connection pools fill. Queues grow. Some requests finish after the rider has already moved. The service is not only slow now. It is spending capacity on work that is no longer useful, while fresh requests wait behind old ones.

Plain meaning:

Backpressure is how a system says, "I cannot accept unlimited work right now. Some callers must slow down, wait with a deadline, receive a smaller answer, or be rejected."

In this scenario:

The ETA service may cap calls to the routing provider, reject some low-value refreshes with Retry-After, serve a cached ETA, or stop accepting requests whose deadline has already passed.

Technical name:

These controls are backpressure. They prevent cascading failure, where one slow component causes waiting, retries, and resource exhaustion across other components.

The Naive Idea: Accept Everything

The first design is simple.

mobile apps -> ETA API -> worker pool -> routing provider

The API receives a request. A worker calls the routing provider. The API returns the ETA.

Under normal load:

arrival rate:       1,000 ETA requests/second
provider capacity:  1,200 ETA responses/second
typical latency:    80 ms

The system has slack. A small burst is fine.

During the city event:

arrival rate:       1,600 ETA requests/second
provider capacity:    500 ETA responses/second
typical latency:     2 seconds
client timeout:      1.5 seconds

The provider can complete fewer requests than the service receives. That mismatch is the important fact.

A queue can smooth a short burst. It cannot create provider capacity. If arrivals stay above completions, the queue becomes a list of promises waiting to expire.

Check: If the ETA API accepts a request and puts it in a queue, has the system protected the user experience?

Think first, then reveal.

Answer: Not necessarily. A queue only helps if the work can still finish before its deadline and if the dependency has enough future capacity to drain the queue. Otherwise the system has accepted a promise that will probably become stale or time out.

The Cascade Starts With Waiting

Slow dependencies are not only slow. They hold resources while they are slow.

A worker waiting on the routing provider may hold:

• a worker slot;
• memory for the request;
• a connection to the provider;
• a trace context and timeout timer;
• sometimes an upstream HTTP connection back to the caller.

When enough requests wait, the ETA service has fewer resources for everything else. The service becomes slow even for requests that might have been cheap.

Now retries arrive.

original user demand:  1,600 requests/second
client retry rate:       700 requests/second
observed API load:     2,300 requests/second

The system did not gain users. It gained repeated attempts from users who already waited too long.

The feedback loop looks like this:

routing provider slows
  -> ETA workers wait longer
  -> API latency rises
  -> clients time out
  -> clients retry
  -> ETA API receives even more work
  -> provider receives more active calls
  -> provider gets slower

This is cascading failure. The original problem may have been one dependency. The cascade is the feedback loop that spreads pressure across callers and services.

A Worked Trace: Break The Feedback Loop

Follow one overload window.

1. The Input Arrives

At 18:00, a concert ends. Many riders open the app.

18:00:00
incoming ETA requests: 1,600/sec
routing provider completion: 500/sec
client timeout: 1.5 sec

Naive decision:

Accept every request.
Start as many provider calls as workers allow.
Let clients retry on timeout.

Intermediate state after one minute:

oldest queued request:  45 sec
active provider calls:  4,000
provider p95 latency:   4 sec
client retries:         rising

Output:

Many riders see spinners. Some get ETAs for old locations. Drivers and riders refresh more often because the app looks stuck.

Naive failure contrast:

The API looked generous because it rejected almost nothing. In practice, it accepted more work than the system could make useful.

2. Put A Deadline On The Promise

An ETA has a short useful life. A route estimate that arrives 45 seconds late may be worse than no estimate. The rider and nearby drivers have moved.

So the request carries a deadline:

eta_request:
  rider_id: rider-9
  driver_id: driver-22
  requested_at: 18:00:02.000
  deadline: 18:00:03.500

The deadline travels with the work. A worker checks the remaining time before calling the routing provider.

if remaining_deadline < estimated_provider_latency:
    do not start provider call
    return cached_or_pending_response

This prevents expired requests from consuming scarce provider capacity.

3. Bound The Queue

The service also bounds waiting work.

queue policy:
  max queued requests: 20,000
  max oldest request age: 2 seconds
  drop expired ETA refreshes

The queue is allowed to absorb a short burst. It is not allowed to hide a long mismatch between arrival and completion.

When the queue age crosses the useful limit, the API changes behavior.

if queue_oldest_age > 2 seconds:
    reject low-value refreshes with 429
    include Retry-After: 3
    serve stale ETA if safe

The rejection is not the failure. The rejection is the system refusing to create a worse failure.

4. Protect The Dependency With A Concurrency Limit

The routing provider performs best with at most 800 active calls from this service. Above that, latency gets worse.

So the ETA service sets a concurrency limit:

provider concurrency limit: 800 active calls

This limit protects the provider from a stampede. It also protects the ETA service from filling its own worker pool with calls that will only make the provider slower.

A concurrency limit is different from a rate limit.

rate limit:        how quickly work may enter
concurrency limit: how much work may be active at once

Both can matter. A rate limit controls arrival. A concurrency limit controls pressure on the dependency.

5. Shed Lower-Value Work

Not every ETA request has the same value.

preserve:
  active trip pickup ETA
  driver navigation update
  rider confirmation after match

shed or degrade:
  map preview while browsing
  background refresh for inactive app
  repeated refresh from the same device

Load shedding means deliberately refusing, delaying, or simplifying lower-value work so the most important promises can survive.

The degraded response might be:

"Routes are updating slowly. Showing last known estimate: 7-9 min."

That response is not perfect. It is bounded, honest, and cheaper than making a new provider call that is unlikely to finish in time.

Check: Why is a stale ETA sometimes safer than accepting a fresh ETA request?

Think first, then reveal.

Answer: If the fresh request cannot finish before its deadline, it will consume scarce capacity and may return too late to be useful. A clearly labeled stale ETA gives the user bounded information while preserving capacity for work that still matters.

6. Make Retries Part Of The Design

Retries are useful for short, random failures. They are dangerous when every client retries at the same time.

The service gives callers a retry policy:

retry only retryable failures
wait with jitter
honor Retry-After
stop at the deadline
limit retry attempts per original request

The goal is to avoid synchronized waves.

Without jitter:

1,000 clients fail at 18:00:01
1,000 clients retry at 18:00:02
1,000 clients retry again at 18:00:04

With jitter and a deadline, retries spread out and stop when the answer would no longer help.

Signals That Tell The Truth

During overload, one metric rarely tells the whole story.

Queue depth matters, but it is not enough. A queue of 10,000 requests might be fine for a batch import and terrible for ETA refreshes.

Useful signals include:

• arrival rate and completion rate;
• oldest queued request age;
• request deadline expiry rate;
• provider active concurrency;
• provider p95 and p99 latency;
• retry attempts per original request;
• rejection rate by priority class;
• stale response rate;
• recovery ramp speed.

The most important comparison is often:

Can the system complete useful work faster than useful work arrives?

If not, accepting more work makes the incident larger.

Common Confusions

Confusion: A Queue Is Capacity

Why it is tempting:

A queue makes the API look calm for a while. Requests have somewhere to go.

Better model:

A queue is a buffer, not a new worker. It buys time during short variation. During sustained overload, it records how many promises are waiting to become stale.

Confusion: Backpressure Means Giving Up

Why it is tempting:

Rejecting or degrading a request feels worse than accepting it.

Better model:

Backpressure is choosing a smaller honest promise over a larger false one. It protects the work that still has a chance to succeed.

Confusion: More Workers Always Help

Why it is tempting:

If work is waiting, more workers sound like the obvious fix.

Better model:

More workers help only when the bottleneck can use more parallelism. If the routing provider is already saturated, more workers increase contention and waiting.

Trade-offs And Limits

Backpressure improves survival under pressure. It does not create infinite capacity.

The central trade-off is success rate versus truthfulness under overload.

If the service accepts every request, the short-term success rate at the API boundary may look high. More callers receive 200 OK. But many of those accepted requests are now fragile promises. They may wait too long, return stale data, time out in the client, or steal capacity from a request that still had a useful deadline.

If the service rejects, delays, or degrades earlier, the API boundary looks less successful. Some users see a controlled failure or a stale estimate. But the system has told the truth about current capacity and preserved the path for higher-value work.

It costs product and operational decisions:

• which requests are highest priority;
• how stale an answer may be;
• when to reject instead of queue;
• what message users should see;
• how quickly to reopen gates after recovery.

It can still fail if priorities are wrong, retry policies are ignored, dependencies do not expose health signals, or degraded responses are misleading.

Recovery also needs backpressure. When the routing provider improves, the ETA service should not instantly open every gate. It should raise admission and concurrency gradually while watching queue age, provider latency, retry rate, and error rate.

recover in steps:
  keep expired work out
  drain high-priority useful work
  raise low-priority admission gradually
  restore background refresh last

The boundary is visible when queue age and deadline expiry keep rising even after rejections begin. That means the system is still accepting or retaining more work than it can make useful.

Practice: Place The Pressure

Take this design:

mobile app -> ETA API -> queue -> workers -> routing provider

Fill in a policy:

maximum useful ETA age:
queue depth limit:
provider concurrency limit:
requests to preserve:
requests to shed or degrade:
client retry rule:
signal that recovery may begin:

Model answer:

maximum useful ETA age: 2 seconds for active matching, 10 seconds for map preview
queue depth limit: enough for a short burst, but reject if oldest request > useful age
provider concurrency limit: the tested level before provider p99 latency rises sharply
requests to preserve: active pickup, matched trip, driver navigation
requests to shed or degrade: inactive background refresh, map browsing, repeated refreshes
client retry rule: honor Retry-After, jittered retry, stop at request deadline
signal that recovery may begin: provider p99 falls, queue age drains, retry rate drops

Now change one parameter: the routing provider returns to normal latency, but clients keep retrying aggressively for five more minutes.

The system should keep retry budgets and admission limits active until observed retries fall. A recovered dependency can be overloaded again by stale client behavior.

Resources

• [BOOK] Site Reliability Engineering: Handling Overload - Focus: Load shedding, graceful degradation, and overload control in production systems.
• [ARTICLE] Using Load Shedding to Avoid Overload - Focus: Practical ways to reject work before saturation becomes collapse.
• [ARTICLE] Timeouts, Retries, and Backoff with Jitter - Focus: Retry behavior, jitter, deadlines, and feedback loops.

Key Takeaways

• Backpressure turns overload into explicit choices: slow down, wait with a deadline, degrade, or reject.
• Queues absorb short bursts, but sustained arrival above completion creates aging promises.
• Cascading failure often comes from a feedback loop: slow dependency, waiting, timeout, retry, more load.
• Concurrency limits protect dependencies; admission control and load shedding protect user-facing promises.
• Recovery should ramp gradually because stale retries and queued work can recreate overload after the original dependency improves.