Rate Limiting and Backpressure for Workers

By the end of this lesson, you will be able to...

• Distinguish a rate limit, a concurrency limit, and backpressure in a worker pipeline.

• Diagnose when a growing queue is protecting a dependency rather than proving workers need to scale.

• Choose an immediate mitigation and a preventive control from queue, dependency, and retry signals.

Idea in one sentence: A healthy worker system sometimes lets work wait on purpose, because safe sustained completion matters more than emptying a queue as fast as possible.

Core Insight

At 09:00, a product launch creates 8,000 video-transcode jobs. The worker pool is healthy. Someone raises its concurrency from four workers to twenty-four so the queue will drain faster.

For several minutes, the graph looks good: queue depth falls. Then the external encoding provider begins returning 429 Too Many Requests. Retry volume climbs. The database receives a burst of job-status updates. Queue depth starts growing again, now full of failed attempts and new uploads.

The workers were not too slow. They were too successful at creating pressure.

backlog -> more claims -> more provider calls -> 429s -> retries -> larger backlog

Lesson 005 established that a pool limits active work. This lesson adds two other controls:

• a concurrency limit bounds how many operations are in flight now;
• a rate limit bounds how quickly new operations may begin over time;
• backpressure is the response when the system recognizes that safe capacity has been reached.

They are not interchangeable knobs. Together, they stop a buffered burst from becoming a downstream outage.

The Production Symptom

At 09:10 the on-call engineer sees:

The tempting diagnosis is “we need even more workers.” The evidence says otherwise. The provider is the active limit, and retries are multiplying work. Adding more consumers would make the provider and database see more failed requests, not more useful completion.

The user impact is not only a long queue. A learner may see “processing” for much longer, urgent jobs can wait behind bulk uploads, and the provider incident lasts longer because the platform keeps sending work it cannot accept.

Three Controls, Three Questions

Plain meaning:

Some controls decide how many jobs can be active. Some decide how fast new jobs can start. Backpressure decides what the system does when those limits are already full.

In this scenario:

The platform permits at most 10 active transcodes, starts at most 30 new provider requests per minute, and stops claiming low-priority jobs when the provider rejects or delays too much work.

Technical names:

Concurrency protects long-lived pressure: CPU, memory, connection pools, and provider sessions. Rate limiting protects burst frequency and quotas. A short request can finish quickly yet still violate a per-minute provider quota. A long request can respect a rate limit yet consume every active slot.

Investigation Path

The on-call response should follow the work path rather than one dashboard number.

queue age rising
-> are workers idle or full?
-> if full, which dependency is slow or rejecting work?
-> are retries increasing the effective arrival rate?
-> which job class must keep moving?
-> apply a temporary bound, then watch useful completion

For the transcode incident:

1. Confirm that workers are full, not dead.
2. Observe the provider's 429 errors and latency; it is the current bottleneck.
3. Lower new starts to the provider-safe rate and cap active encodes.
4. Reduce retry pressure with delayed, bounded attempts rather than immediate requeueing.
5. Reserve a small path for urgent instructor fixes, or defer bulk re-encodes.
6. Watch provider acceptance, completion rate, oldest-job age, and database latency together.

The mitigation may make the queue drain more slowly at first. That is expected. The goal is to restore stable useful throughput: jobs that complete without producing new failed work.

A Worked Control Trace

Assume the provider accepts 30 starts per minute and can sustain 10 active encodes. Jobs average two minutes.

One simplified admission check makes the two budgets visible:

def may_start_transcode(in_flight, limiter):
    if in_flight >= 10:
        return False
    return limiter.try_take_token()  # at most 30 starts per minute

This is not a complete distributed limiter implementation. The important point is that one check protects active work and the other protects start frequency. If either refuses work, the worker does not turn that refusal into an immediate retry storm.

The naive failure contrast is:

naive claim: an empty queue proves the system is healthy
better claim: completed useful work within dependency limits proves the pipeline is healthy

So far, the queue is allowed to hold work. It is not abandoned. Its age, priority, and drain rate are now part of an explicit product and operational contract.

The Failure Mechanism: Retries Can Defeat a Limit

Suppose every provider 429 is immediately requeued. The rate limiter may allow 30 first attempts per minute, but failed work returns at the same time as new work. Without delayed and bounded retries, the effective arrival rate rises just when capacity fell.

provider rejects
-> retry immediately
-> more claims compete with fresh jobs
-> more provider rejects
-> work multiplies without completion

Backpressure changes that loop. It can pause claims briefly, delay retries, reduce the producer's nonessential job creation, and protect a priority lane. The exact retry and dead-letter semantics are developed later in the track. Here, the operational rule is enough: failed work must not bypass the same safety budget that protected the first attempt.

Mitigation and Prevention

Immediate mitigation

• Lower transcode concurrency to the provider's active-session budget.
• Apply a start-rate limit below the observed rejection threshold.
• Delay retries and cap their number; do not requeue immediately on 429.
• Pause or defer bulk low-priority work.
• Preserve capacity for jobs with a tighter user promise.

Prevention

• Store per-dependency limits as reviewed policy, not defaults hidden in worker code.
• Separate job classes when priorities or downstream limits differ.
• Alert on provider rejection, retry growth, oldest-job age, and completion collapse together.
• Test degraded dependency behavior with a bounded burst before the next launch.

This is not a call to keep limits permanently low. It is a call to increase them only with evidence that the limiting dependency can sustain the extra pressure.

Signals to Watch

One chart rarely explains worker overload. Use a small set that connects cause and outcome:

• queue depth and oldest-job age: how much work waits and how long users wait;
• active claims: whether the pool is full, idle, or stuck;
• start rate and limiter rejections: whether the chosen budgets are binding;
• provider latency and 429/error rate: whether the downstream system is the limit;
• completion rate: whether work actually finishes;
• retry rate and delayed-retry backlog: whether failures are amplifying demand;
• priority-class age: whether protective controls accidentally starve urgent work.

A good alert combines a symptom with a time budget. “Queue has 1,000 jobs” is vague. “Oldest reminder exceeds 10 minutes while completion rate falls and provider rejections rise” tells an operator which path is failing and why it matters.

Trade-offs and Limits

The central trade-off is immediate drain speed versus dependency safety and recovery. A tighter limit makes some jobs wait longer. It also prevents failed starts, retries, and cascading latency from consuming more capacity than the successful work.

Backpressure is useful when the system can postpone or prioritize work. It does not make an indefinitely overloaded service meet every deadline. If arrival rate remains above sustainable completion for long enough, the team needs a product decision, more safe downstream capacity, or admission control at the producer boundary.

Limits can fail too. A global counter might be unavailable, a threshold may be too strict, or an emergency priority lane can starve normal work if it becomes the default. You see the boundary when oldest-job age keeps climbing despite stable limits, provider errors stay high under the safe rate, or one class never receives its share of capacity.

Common Confusions

Confusion: Rate and concurrency limits are the same control

Why it is tempting:

Both can make workers start fewer jobs.

Better model:

Concurrency limits active work now. Rate limits starts over time. Long jobs can exhaust concurrency without exceeding a rate; fast jobs can exceed a rate without many active operations.

Confusion: A long queue proves the system is broken

Why it is tempting:

Waiting work is visible and uncomfortable.

Better model:

A queue can be a healthy buffer during a finite burst. Evaluate its age, priority, and drain behavior alongside dependency signals. An empty queue created by rejected work is worse than a bounded queue that completes steadily.

Confusion: Backpressure means stop all work

Why it is tempting:

“Slow down” sounds like a global pause.

Better model:

Backpressure can be selective: reduce one dependency's starts, defer low-priority jobs, or reserve capacity for urgent work. The response should match the scarce resource and the product promise.

Check Your Understanding

Check: A provider permits 30 starts per minute. Jobs now finish in five seconds, and workers begin 80 jobs per minute while only three are active at any moment. Which control is missing?

Think first, then reveal.

Answer: A rate limit. Low active concurrency does not prevent too many fast starts within the provider's minute-level quota.

Check: Queue depth rises during a launch, provider errors are zero, completion rate remains stable, and oldest-job age stays below the promised delay. Must the team add workers immediately?

Think first, then reveal.

Answer: No. The queue may be absorbing a finite burst safely. Add capacity only after checking whether the sustained arrival rate and user delay budget require it.

Check: 429 errors and retries spike after raising worker concurrency. What is the safer first mitigation?

Think first, then reveal.

Answer: Reduce active starts to the provider's safe budget, delay retries, and stop low-priority claims. Adding more workers would increase the rejected work path.

Practice

An email provider accepts 100 sends per minute and 20 active connections. Your platform has password resets, class reminders, and a bulk marketing campaign in one worker queue. During the campaign, password resets are delayed.

Propose a control policy. Include:

• the concurrency and start-rate budget;
• how you separate or prioritize the three job classes;
• the backpressure response when provider rejections rise;
• two signals that decide whether the intervention helped;
• one failure boundary the policy does not solve.

A strong answer reserves capacity for password resets, gives marketing a paced low-priority path, delays rather than immediately retries rejected sends, and measures urgent-job age plus provider errors and useful completion. It acknowledges that persistent demand beyond provider capacity needs a product or provider-capacity decision.

Resources

• [ARTICLE] AWS: Exponential Backoff and Jitter — Focus: connect retry timing to load shaping and recovery.
• [DOC] Stripe Rate Limits — Focus: see how an external dependency exposes rate and concurrency constraints.
• [DOC] RabbitMQ Consumer Prefetch — Focus: relate bounded in-flight delivery to worker pressure control.

Key Takeaways

1. Concurrency, rate, and backpressure answer different questions. Use each to control active pressure, start frequency, and overload response.
2. A queue can be healthy waiting work. Judge it by age, priority, completion, and dependency health—not by depth alone.
3. Retries are load. A failed attempt must obey the same pressure budget as a new one.
4. Sustainable completion is the goal. A slower stable pipeline beats a fast path that creates rejection and cascades.