Backpressure, Lag, and Operational Signals

By the end of this lesson, you will be able to...

• Interpret lag, throughput, retry, and saturation signals in an event-driven system.

• Trace where pressure builds across producer, broker, and consumer stages.

• Choose a mitigation that protects the user promise instead of only hiding the metric.

Idea in one sentence: Lag is not the problem by itself; it is evidence that work is arriving, waiting, failing, or being retried somewhere faster than the system can safely finish it.

Core Insight

An order platform publishes OrderPaid.

Three consumers react:

fulfillment-service -> starts packing
email-service       -> sends receipt
analytics-service   -> updates revenue projection

At 09:00, a promotion begins. Order volume triples.

At 09:15, the dashboard shows this:

A teammate says:

The queue is the problem.

Maybe. But maybe not.

The broker is where waiting becomes visible. The cause may be somewhere else: a slow email provider, too much producer traffic, a consumer bug, a retry storm, a bad partition key, or a downstream database that cannot accept writes quickly enough.

Backpressure is the system's way of saying:

Slow down, drop lower-value work, shed load, or change the path.
I cannot safely finish all of this at the current rate.

Lag is the visible backlog. It tells you that published work has not yet been completed by some consumer. It does not tell you why.

The operational skill is to read the signals together.

The Naive Reaction

The naive reaction is:

Lag is high.
Add more consumers.

Sometimes that is correct.

If each email worker can send 50 receipts per minute and the provider allows more parallel requests, adding workers may reduce lag safely.

But adding workers can also make the incident worse:

• the email provider may rate-limit harder
• retries may multiply the work
• a downstream database may saturate
• partitions may keep most work on one hot key
• duplicate side effects may rise if idempotency is weak
• a backfill may compete with live traffic

The queue is only one part of the path.

Plain meaning:

Backpressure means pressure travels backward through the system. A slow stage forces earlier stages to buffer, slow down, reject, or reshape work.

In this scenario:

The email provider accepts only 300 receipt sends per minute. The order service publishes 900 paid orders per minute. The email consumer cannot finish work as fast as it receives it, so lag grows.

Technical name:

The waiting work is consumer lag or queue depth, depending on the broker model. The mechanism that prevents unlimited waiting is backpressure. The evidence you use to decide what to do is operational signals: rates, saturation, retries, errors, age of oldest message, DLQ movement, and user impact.

What Each Stage Can See

Make the path visible before choosing a fix.

producer -> broker/log -> consumer -> dependency -> side effect

Each stage sees a different part of the truth:

This is why one graph is rarely enough.

High lag with healthy processing may mean the system is catching up after a planned backfill.

High lag with rising retries may mean the same events are failing again and again.

High lag with low consumer CPU may mean workers are waiting on an external dependency.

High lag on one partition may mean the partition key put too much work behind one customer, tenant, or merchant.

A Worked Incident Trace

Return to the promotion incident.

The team has these facts:

09:00 promotion starts
09:05 email provider latency rises from 120 ms to 1.8 s
09:08 email consumer timeout stays at 1 s
09:10 timeout retries begin
09:15 email lag reaches 18,000 events
09:20 customers ask where receipts are

Now trace the pressure:

The important intermediate state is step 2.

The consumer timed out, but the provider may still have accepted some emails. That means retries are not just extra work. They may be duplicate side effects.

The better operational question is:

What work should continue now, what work should slow down, and what work needs idempotency before retry?

So far:

lag tells us work is waiting
retry rate tells us some work is not completing cleanly
dependency latency tells us where the wait starts
oldest message age tells us user-visible delay
idempotency records tell us whether retry is safe

Check: In the trace above, why is "add more email consumers" risky before checking provider limits and idempotency records?

Think first, then reveal.

Answer: More consumers can create more parallel calls to the same slow provider. If some timed-out calls actually succeeded, more retries can create duplicate receipts unless the consumer has a stable idempotency key or sent-receipt record.

Reading Common Signals

Use signals as clues, not verdicts.

This table keeps the diagnosis close to the evidence.

Lag alone says:

not done yet

The surrounding signals say:

too much input
too little capacity
bad retry behavior
blocked partition
unsafe dependency
poison event
recovery work competing with live work

Choosing a Mitigation

A mitigation should match the pressure point.

If producers are creating too much low-value work, slow or filter the producers. For example, collapse repeated CartViewed events into a sampled analytics stream.

If a consumer is slow but safe to parallelize, add workers or partitions. This helps only when the dependency and ordering model allow parallelism.

If retries are amplifying failure, add backoff, retry budgets, or a circuit breaker. A retry budget says: after a limit, stop retrying through the hot path and move the event to a controlled recovery path.

If one partition is hot, review the partition key. A key like merchant_id may serialize all work for one large merchant. A better key may spread work while preserving the ordering scope the business actually needs.

If live traffic and backfill traffic compete, separate their lanes. Recovery work may need lower priority, a smaller window, or a shadow projection so live user-facing work stays healthy.

If a side effect is uncertain, do not blindly retry. First find the idempotency key, inbox record, provider request id, or sent-effect table that says whether the effect already happened.

The decision should protect the user promise.

For the email incident, the promise might be:

Customers should receive a receipt eventually.
They should not receive five copies.
Fulfillment should not stop just because receipts are delayed.

That promise suggests a concrete response:

1. Keep OrderPaid publishing, because the fact is true and fulfillment needs it.
2. Rate-limit email sending to provider-safe throughput.
3. Use order_id + receipt_type as the idempotency key before re-sending.
4. Move events that exceed retry budget to a receipt recovery queue.
5. Alert on oldest unsent receipt age, not only total lag.

Trade-offs and Limits

Backpressure improves safety because it prevents a slow stage from silently creating unlimited work.

It costs latency, complexity, or dropped work.

If you throttle producers, users may see slower acceptance or rejected requests. If you buffer more, users may see delayed side effects. If you add consumers, dependencies may saturate. If you drop low-value events, analytics may become approximate. If you split traffic into priority lanes, operations becomes more complex.

Backpressure does not tell you the business priority of work. The system may know that two events are waiting. It does not know that one receipt affects a customer and one analytics click can be sampled unless you design that policy.

Lag also has limits as a metric.

A topic with 1,000 tiny events may be healthier than a topic with 10 old events that always fail. A count can hide age. An average can hide one stuck partition. A successful consumer ack can hide a side effect that was skipped incorrectly.

You can see the boundary when:

• lag falls but duplicate side effects rise
• consumer throughput rises but provider errors rise too
• queue depth is low but oldest message age remains high
• backfill completes but live traffic misses its delay target
• retry count is high but successful user outcomes do not improve

The trade-off is direct:

Event-driven systems buy time by putting work between services.
That time becomes useful only when the system also has policies for slowing, prioritizing, retrying, and observing the work.

Common Confusions

Confusion: Lag means the broker is broken

Why it is tempting:

The backlog is visible in the broker dashboard, so the broker feels like the failing component.

Better model:

The broker is often the pressure gauge. The cause may be producer volume, consumer speed, dependency limits, retry behavior, partitioning, or poison events.

Confusion: More consumers always reduce lag

Why it is tempting:

If one worker is slow, more workers sound like more capacity.

Better model:

More consumers help only when work can be parallelized and downstream dependencies can handle the extra concurrency. They can worsen rate limits, duplicate side effects, and retry storms.

Confusion: Backpressure is just an infrastructure concern

Why it is tempting:

Backpressure often appears as queue depth, broker storage, worker pools, or stream-processing metrics.

Better model:

Application architecture decides which work may wait, which work may be dropped, which side effects must be protected, and which user promise matters most. Infrastructure exposes the pressure, but product and service boundaries define the response.

Practice

Review this incident:

The billing service publishes InvoiceReady.
The PDF service consumes the event, generates an invoice PDF, stores it, and sends InvoicePdfCreated.
A new enterprise customer imports 80,000 invoices.
PDF consumer lag rises from 200 to 60,000.
CPU on PDF workers is 95%.
The object store is healthy.
One customer support page waits for InvoicePdfCreated before showing invoice status.
Analytics also consumes InvoiceReady, but it is healthy.

Choose a response plan.

A good answer should mention:

• The lag is scoped to PDF generation, not all event processing.
• The import is a burst of expensive work; CPU saturation suggests the PDF workers are the bottleneck.
• Adding workers may help if PDF generation can be parallelized and ordering is not required per invoice.
• The support page should not depend on immediate PDF completion for basic invoice visibility; it can show "PDF pending" from source billing state.
• Batch/import work may need a lower-priority lane so normal invoices do not wait behind 80,000 imported invoices.
• Alerts should include oldest pending PDF age and user-visible delay, not only total lag.
• The system should avoid retrying failed PDF jobs without classifying whether failures are temporary, poison inputs, or capacity pressure.

Resources

• [BOOK] Martin Kleppmann - Designing Data-Intensive Applications
  • Link: https://dataintensive.net/
  • Focus: Use the chapters on logs, messaging, and derived data to connect backlog with system design.
• [ARTICLE] Martin Fowler - What do you mean by Event-Driven?
  • Link: https://martinfowler.com/articles/201701-event-driven.html
  • Focus: Revisit the event styles and ask which side effects and user promises each style creates.
• [DOC] Kafka consumer lag monitoring
  • Link: https://docs.confluent.io/platform/current/monitor/monitor-consumer-lag.html
  • Focus: Treat lag as an operational signal that needs context from throughput, partitions, and consumer health.
• [DOC] Google Cloud Pub/Sub flow control
  • Link: https://cloud.google.com/pubsub/docs/flow-control-messages
  • Focus: Use flow control as a concrete example of limiting outstanding work instead of accepting unbounded pressure.

Key Takeaways

• Lag means work is waiting, but it is not a diagnosis by itself.
• Backpressure is the mechanism or policy that prevents a slow stage from creating unlimited unsafe work.
• Operational diagnosis needs multiple signals: publish rate, consumer rate, oldest message age, retries, errors, partition skew, dependency latency, and user impact.
• Adding consumers helps only when parallelism is safe and downstream dependencies can handle it.
• Good event-driven architecture defines which work may wait, which work may slow down, which work may be dropped, and which side effects must be protected.