Reliability Control Loops and Feedback

By the end of this lesson, you will be able to...

• Trace a reliability control loop from user promise to signal, decision, action, and learning.

• Identify the control surface, feedback delay, and next observation for one service.

• Explain how a local reliability action can improve one service while hurting the wider system.

Idea in one sentence: Reliability improves when signals are connected to decisions and actions, then checked again after the system has had time to respond.

Core Insight

The checkout team launched the new payment flow after a production-readiness review.

The review was useful. It named the promise:

Users should receive paid, declined, or safely pending within 2 minutes.
Users should not be charged twice for one checkout attempt.

It also named the signals:

clear-result SLI
pending-payment queue age
duplicate-prevention hits
payment-provider timeout rate
release state

Now the service is live.

At 10:20, the clear-result SLO starts burning fast. Pending-payment queue age rises. The payment provider is slow. A new adapter version was deployed earlier in the morning.

The naive reaction is:

We have dashboards and alerts, so we have feedback.

Not quite.

A dashboard is a sensor. An alert is a signal. Neither one is a full feedback loop by itself.

A reliability control loop exists only when the team can connect:

promise -> signal -> comparison -> decision -> action -> new observation -> learning

If the loop is broken, the team may stare at correct data and still act too late, act twice, or act in a way that helps checkout while damaging another part of the system.

Plain meaning:

Feedback means the system tells you what changed after an action.

In this scenario:

Checkout opens a circuit breaker, rolls back a deploy, caps reconciler retries, and then checks whether clear-result burn and queue age actually improve.

Technical name:

That repeated path from signal to action to new signal is a reliability control loop.

The Small Situation

Use the same checkout payment flow from the previous lessons.

The important actors are:

user
checkout service
payment provider
payment reconciler
on-call engineer
release owner
support team

The promise is user-facing:

The user gets a truthful result within 2 minutes:
paid, declined, or safely pending.

The system has a degraded mode:

If the payment provider is too slow, checkout records payment_pending,
shows an honest pending message, and lets the reconciler finish later.

The system also has possible actions:

open payment-provider circuit
pause checkout promotion
roll back payment-adapter version
cap reconciler retries
add worker capacity
route support to affected order IDs

Those actions are the control surface.

Plain meaning:

Control surface means the knobs, switches, limits, and decisions the team can actually change.

In this scenario:

The SLO is not a control surface. The SLO is a target. The rollback button, circuit breaker, retry cap, traffic limit, and degraded-mode policy are control surfaces.

Technical name:

In control-loop language, these actions are actuators. They are the parts that can change the running system.

The Naive Loop

A weak loop looks like this:

1. Alert fires.
2. On-call opens dashboard.
3. On-call sees provider timeouts.
4. On-call increases retries.
5. Queue age keeps rising.
6. On-call increases retries again.

This feels active.

It is also risky.

The team is acting before it has a clear target, a named control surface, or a feedback delay.

The missing questions are:

What user promise is failing?
Which signal proves it?
What target are we comparing against?
What action can change the system?
How long should that action take to show an effect?
What new signal will tell us whether the action helped?
What harm could the action create outside checkout?

Without those questions, "responding quickly" can become overcorrecting quickly.

The most common version is retry amplification.

Checkout sees slow provider responses. The local queue grows. The team increases retries to drain the queue. Each checkout worker now sends more requests to the same unhealthy provider. The provider gets slower. Other services that use the same provider get worse too.

Locally, the action sounded reasonable.

Globally, it made the system less reliable.

The Mechanism Step by Step

A useful reliability control loop has six visible parts.

Now put those parts in motion:

1. Promise
   Users should get a clear checkout result within 2 minutes.

2. Signal
   clear_result_ratio drops and pending queue age rises.

3. Comparison
   The SLO is burning at 12x normal rate.
   Queue age is above the degraded-mode target.

4. Decision
   The service cannot afford extra change risk.
   Keep degraded mode active, pause promotion, and roll back the adapter.

5. Action
   Open the provider circuit for user-path calls.
   Cap reconciler retries.
   Roll back adapter version 42.

6. New observation
   Wait one feedback delay.
   Then check clear-result burn, queue age, duplicate-prevention hits,
   provider timeout rate, and support contacts.

7. Learning
   If queue age falls and clear-result burn slows, continue controlled recovery.
   If queue age rises, escalate and change the control action.

This is not only incident response.

It is also learning. The team updates its model of the service based on what happened after the action.

A Worked Trace

Here is one trace.

Starting state:

10:00  adapter v42 is serving 20% of checkout traffic
10:04  payment-provider p95 latency rises above 8 seconds
10:12  pending-payment queue age reaches 6 minutes
10:20  clear-result SLO burn reaches 12x
10:21  duplicate-prevention hits are 4x baseline

Naive decision:

Provider is slow.
Increase reconciler retries from 3 to 10.

Why that can fail:

More retries create more provider calls.
The provider is already slow.
Checkout queue age may fall briefly, but provider health may get worse.
Other services sharing the provider may also slow down.

Better control-loop trace:

The important part is the delay.

Some actions need time before the signal changes. A rollback needs deployment time. A queue needs drain time. A support message needs time before support contacts change.

If the team changes another knob every minute, it cannot tell which action caused which result.

It also risks oscillation:

too many retries -> provider worse -> retries reduced -> queue worse -> retries increased again

The loop becomes noisy instead of useful.

So far, the mechanism is:

Use the promise as the target.
Use user-centered signals as sensors.
Compare against the target.
Choose one bounded action.
Wait long enough to see the effect.
Then update the operating model.

Feedback Delay

Feedback delay is the time between action and useful evidence.

It is not always the same as alert delay.

Alert delay:

How long until we notice something is wrong?

Feedback delay:

How long until we can tell whether our action helped?

For checkout:

Opening the circuit may affect user latency in seconds.
Rolling back the adapter may affect error rate in minutes.
Capping retries may affect provider health in minutes.
Draining pending payments may take tens of minutes.
Reducing support contacts may take longer.

A good loop names the expected delay before the action.

Example:

Action:
  cap reconciler retries at 3 and keep provider circuit open

Expected feedback delay:
  15 minutes for queue-age direction

Next observation:
  queue age should stop increasing and then begin falling

Escalation condition:
  if queue age is still increasing after 15 minutes, page provider owner
  and reduce local reconciliation pressure further

This keeps the team from confusing impatience with evidence.

Check: Why is it risky to change retry limits every minute during a provider incident?

Think first, then reveal.

Answer: The system has not had time to show the effect of the last action. Changing the limit again can hide cause and effect, create oscillation, and amplify load on the provider.

Local and Global Effects

Reliability actions have a scope.

A local action can improve one service while hurting the system around it.

Examples:

This is why the loop must include the plant: the real system that responds.

Checkout is not only checkout code. It includes provider limits, queues, database capacity, support flow, user behavior, and release state.

A control loop that ignores the wider plant can optimize the wrong boundary.

The next lesson will make this even more explicit: reliability targets are also economic and product decisions. For now, the key point is simpler:

Before acting, ask what boundary the action protects.
After acting, watch for harm outside that boundary.

Check: In the checkout scenario, is the SLO a control surface?

Think first, then reveal.

Answer: No. The SLO is the target or reference. A control surface is something the team can change, such as traffic percentage, rollback state, circuit breaker state, retry limits, or degraded-mode behavior.

Trade-offs and Limits

Control loops improve reliability because they connect evidence to action.

They help when:

signals are tied to a user promise
actions are known before pressure arrives
owners know who decides
the feedback delay is named
the next observation is explicit

They cost design and operational attention.

Someone must decide:

which signals matter
which thresholds trigger action
which actions are safe
which actions automation may take
which actions require human judgment

Fast loops can reduce user impact quickly.

They can also become unstable if the signal is noisy or the action is too strong.

Slow loops are less likely to overreact.

They can also let error budget burn for too long.

That is the central trade-off: fast feedback protects users sooner, but it can amplify noise; slower feedback is steadier, but it can leave users in a bad state longer.

Automation can help when the condition and action are well understood:

if provider timeout rate crosses threshold,
open the user-path circuit for 5 minutes

Automation is dangerous when the signal is ambiguous or the action has wide blast radius:

if checkout latency rises,
increase retries everywhere

The boundary signal is repeated manual override.

If humans constantly disable the automation, the loop is probably using the wrong signal, wrong threshold, wrong action, or wrong delay.

Common Confusions

Confusion: "A dashboard is a feedback loop"

Why it is tempting:

A dashboard shows changing data, so it feels like feedback.

Better model:

A dashboard is a sensor. It becomes part of a feedback loop only when it connects to a target, a decision, an action, and a new observation.

Confusion: "More feedback is always better"

Why it is tempting:

More data feels safer than less data.

Better model:

Useful feedback has the right signal, timing, and action. Noisy or too-frequent feedback can cause overreaction.

Confusion: "Automation removes humans from reliability"

Why it is tempting:

Automation can act faster than people.

Better model:

Humans still design the target, thresholds, allowed actions, safety limits, and review process. Automation moves the action; humans own the policy.

Confusion: "If checkout improved, the action was globally good"

Why it is tempting:

Teams naturally watch their own service first.

Better model:

A local improvement can push cost into dependencies, support, databases, or other services. Watch the boundary around the action, not only the local SLI.

Practice

Trace this new control loop.

Scenario:

Search API promise:
  users should receive useful results or a clear degraded response within 1 second

Signals:
  p95 search latency is 2.8 seconds
  cache hit rate dropped from 82% to 35%
  database read pool is saturated
  error budget burn is 8x

Possible actions:
  disable expensive ranking feature
  increase database read pool
  serve cached popular results
  increase client retries
  roll back the last ranking deploy

Write a control-loop note with:

1. Target promise.
2. Main sensor.
3. Comparator result.
4. Chosen control surface.
5. Expected feedback delay.
6. Next observation.
7. One local/global risk.

Model answer:

Target:
  useful results or clear degraded response within 1 second

Main sensor:
  p95 latency, cache hit rate, database read-pool saturation, error-budget burn

Comparator:
  latency is above target and burn is 8x, so the promise is currently failing

Chosen control surface:
  disable expensive ranking and serve cached popular results

Expected feedback delay:
  a few minutes for p95 latency and database saturation to move

Next observation:
  p95 should fall toward 1 second, read-pool saturation should drop,
  degraded-response rate may rise but should be honest and bounded

Local/global risk:
  increasing client retries might look helpful for individual requests,
  but it can add load to an already saturated database path

A good answer does not need to choose the same action, but it must connect the action to the promise, name the delay, and watch for second-order harm.

Resources

• [BOOK] Site Reliability Engineering: Service Level Objectives
  • Link: https://sre.google/sre-book/service-level-objectives/
  • Focus: Use it to connect control loops to explicit user-centered targets.
• [BOOK] Site Reliability Engineering: Monitoring Distributed Systems
  • Link: https://sre.google/sre-book/monitoring-distributed-systems/
  • Focus: Read for the difference between collecting signals and supporting action.
• [BOOK] Site Reliability Engineering: Addressing Cascading Failures
  • Link: https://sre.google/sre-book/addressing-cascading-failures/
  • Focus: Pay attention to retry amplification, overload, and local actions that create wider failure.
• [BOOK] Site Reliability Workbook: Alerting on SLOs
  • Link: https://sre.google/workbook/alerting-on-slos/
  • Focus: Use it to think about when a signal should trigger a human or automated response.

Key Takeaways

• A reliability control loop connects a user promise to signals, comparison, decision, action, new observation, and learning.
• Dashboards and alerts are sensors, not complete feedback loops by themselves.
• The control surface is what the team can change: rollback, traffic, circuit breakers, retry caps, degraded modes, or capacity.
• Feedback delay matters because acting again too soon can hide cause and effect or create oscillation.
• Local reliability actions can create global harm, so every loop should watch the boundary around the action.