Reliability Synthesis Review

By the end of this lesson, you will be able to...

• Connect reliability promises, indicators, objectives, budgets, alerts, degraded modes, readiness, control loops, economics, and anti-patterns.

• Review a service reliability plan and identify the missing decision or evidence.

• Write a short reliability review memo that prepares a service for the final capstone.

Idea in one sentence: Reliability engineering is the practice of turning a user promise into measurable evidence, operating decisions, safe actions, and explicit trade-offs.

Core Insight

The track started with a simple question:

What does it mean for a service to be reliable?

The tempting answer was:

The service is up.

That answer is too small.

Users do not experience "up" as an internal state. They experience outcomes:

Did checkout give me a clear result?
Was I charged once?
Can I see my order?
Did support know what happened?
Did the system recover without making things worse?

Across the track, we replaced the simple model with a better one:

promise -> indicator -> objective -> budget -> decision -> action -> feedback -> learning

That model is the center of reliability engineering foundations.

Every lesson added one piece:

Promise:
  what users need from the service

SLI:
  how we measure the promise

SLO and error budget:
  how we decide acceptable imperfection

Trade-off:
  what gets better and what gets harder

Alert:
  when a human should act

Degraded mode:
  what smaller promise we can still keep

Incident handoff:
  what evidence moves to the next owner

Release safety:
  when change should slow down

Readiness review:
  whether the service can make and operate the promise

Control loop:
  how signals become actions and learning

Reliability economics:
  how product value and cost shape the target

Anti-pattern diagnosis:
  how we detect practices that look mature but lack mechanism

This lesson is a synthesis review.

It does not add a large new idea.

It helps you hold the existing ideas together so the capstone is not just a checklist.

What You Can Now See

Before this track, a service outage might look like one problem:

Checkout is broken.

Now you can separate the problem into sharper questions:

Which user promise is failing?
Which SLI proves it?
Which SLO or budget tells us how urgent it is?
Which reliability dimension is under pressure?
Which dependency or capacity limit is involved?
Which degraded mode protects users now?
Which alert should page a human?
Which release decision should change?
Which evidence belongs in the handoff?
Which action is available?
How long before feedback should appear?
What cost or risk are we accepting?
Which anti-pattern should we avoid?

That is the practical value of the track.

It turns a vague reliability conversation into an inspectable operating conversation.

Here is the most important distinction:

Reliability artifact:
  dashboard, alert, SLO, checklist, runbook, policy

Reliability mechanism:
  user promise, signal, comparison, owner, action, feedback, trade-off

Artifacts are useful only when they serve the mechanism.

The Concepts Together

Use this map when reviewing a service.

This map is not a form to fill blindly.

It is a sequence of pressure tests.

If one row is weak, the service may still be operable. But the weakness should be visible.

Hidden weakness creates surprises.

Visible weakness creates decisions.

Common Confusions

Confusion: "The SLO is the reliability plan"

Why it is tempting:

The SLO looks formal. It has a number, a window, and a target.

Better model:

The SLO is a decision boundary. The reliability plan also needs signals, alerts, degraded modes, release rules, ownership, readiness evidence, and a cost/risk judgment.

Confusion: "Incidents start only after everything is failing"

Why it is tempting:

Teams often wait for visible user complaints before treating a situation seriously.

Better model:

Incidents often start as budget burn, queue growth, dependency slowdown, retry amplification, or ambiguous user state. Early evidence matters because it gives the team a chance to reduce harm.

Confusion: "Degraded mode means lower quality, so it is bad"

Why it is tempting:

Degraded mode sounds like giving users a worse product.

Better model:

A degraded mode keeps a smaller truthful promise when the full promise is unsafe. "Safely pending" is better than a fake success, duplicate charge, or endless spinner.

Confusion: "The final answer is always stricter reliability"

Why it is tempting:

Higher reliability numbers look safer.

Better model:

The right target depends on user harm, cost, product value, and accepted risk. Some promises deserve extreme reliability. Others deserve honest degradation and clear support workflows.

Confusion: "Anti-pattern diagnosis is an extra step"

Why it is tempting:

It can feel like a review after the review.

Better model:

Anti-pattern diagnosis is how you test whether the reliability artifacts actually work. It keeps dashboards, alerts, SLOs, checklists, and automation connected to the user promise.

Synthesis Example

Review this checkout reliability plan.

Starting plan:

Service:
  checkout payment flow

Promise:
  checkout should be reliable

Dashboard:
  HTTP 500s, CPU, memory, pod restarts, provider timeout rate

SLO:
  99.99% availability

Alerts:
  page on CPU above 80%
  page on any pod restart
  page on provider timeout rate above 2%

Degraded mode:
  retry payment provider until success

Readiness:
  dashboard, alert, owner, runbook, and rollback all marked yes

Release policy:
  releases continue if tests pass

Economics:
  product wants best possible reliability

At first, this looks organized.

But it has the anti-patterns from the previous lesson:

promise is vague
SLI does not measure user outcome
SLO is decorative
alerts wake humans without user-impact action
degraded mode can amplify provider failure
readiness checks possession instead of evidence
release policy ignores budget burn
economics hides cost and accepted risk

Now rewrite it as a short reliability review memo.

Reliability Review Memo

Service:
  checkout payment flow

User promise:
  users receive paid, declined, or safely pending within 2 minutes
  one checkout attempt does not create duplicate charges

Primary SLIs:
  clear-result ratio within 2 minutes
  duplicate-risk signal
  pending-payment queue age

SLO and budget:
  99.9% clear result within 2 minutes over 30 days
  duplicate-charge prevention treated as a stricter correctness guard
  fast burn pauses non-mitigating checkout releases

Important trade-off:
  safely pending is worse than immediate paid or declined,
  but it is better than fake success, duplicate charge, or endless waiting

Alerting:
  page on fast burn of clear-result SLO
  page on duplicate-risk signal
  route provider timeout to dashboard unless user impact appears

Degraded mode:
  provider timeout writes payment_pending with idempotency key
  user sees honest pending message
  reconciler retries with caps
  support can inspect pending state safely

Capacity and dependency signals:
  pending queue age
  reconciler worker saturation
  provider timeout rate
  duplicate-prevention hits

Incident handoff:
  impact, timeline, current mitigation, unknowns,
  queue age, affected order IDs, owners, next review time

Release safety:
  pause checkout promotion during fast burn
  rollback only if old code can read pending-payment state
  allow targeted mitigations that reduce current risk

Readiness evidence:
  rollback compatibility tested
  degraded mode tested with provider timeout
  support workflow tested
  alert includes first action and escalation condition

Control loop:
  signal -> compare to SLO -> decide -> open circuit/cap retries/pause release
  wait expected feedback delay
  observe queue age, clear-result burn, duplicate-risk signal, support contacts

Economics:
  buy 99.9 now with support tooling, retry caps, and release discipline
  do not buy active-active provider failover yet
  revisit if enterprise risk, churn, or support volume crosses threshold

Anti-pattern check:
  no paging on isolated CPU
  no SLO without policy
  no readiness item without evidence
  no retries without caps and feedback delay

This memo is not long.

It is useful because each line connects to a decision.

It also shows the central trade-off:

The team buys enough reliability to protect checkout trust,
but it does not buy every expensive protection immediately.

That trade-off is explicit, owned, and reviewable.

So far, the synthesis pattern is:

name the promise
measure the promise
set the target
decide what happens when the target is under pressure
define the safe degraded behavior
connect alerts to actions
test readiness with evidence
run a feedback loop
make cost and accepted risk visible
check for anti-patterns

Retrieval Check

Use these questions to test whether the pieces are connected.

Check: A team has a provider-timeout alert but no clear-result SLI. What concept is missing?

Think first, then reveal.

Answer: The user-centered SLI is missing. Provider timeout may explain failure, but the team still needs to measure whether users receive a truthful checkout result within the promised window.

Check: A service is burning budget quickly, but releases continue because tests passed. Which prior idea should change the decision?

Think first, then reveal.

Answer: Release safety. Passing tests is not enough when current reliability evidence says the service cannot afford extra change risk.

Check: A degraded mode shows "success" before payment is confirmed. Which reliability dimension is at risk?

Think first, then reveal.

Answer: Correctness and truthfulness are at risk. The service may improve perceived latency, but it can mislead users and create payment ambiguity.

Transfer Challenge

Review this order-history plan.

Service:
  order history

Current claim:
  order history is reliable because API uptime is 99.95%

Known user pain:
  recent orders sometimes disappear for 20 minutes during dependency lag
  support contacts spike when receipts are delayed
  payment correctness is safe

Dashboard:
  API uptime, CPU, memory, request count

Alert:
  page on API 500 rate above 1%

Current degraded behavior:
  show an empty order list

Release policy:
  deploy if tests pass

Product view:
  delayed receipts are painful, but not as severe as payment errors

Write a short reliability review memo.

Include:

1. User promise.
2. SLI and SLO.
3. Budget or release decision.
4. Degraded mode.
5. Alerting or handoff.
6. Cost/risk trade-off.
7. Anti-pattern to avoid.

Model answer:

User promise:
  users can see recent orders and receipts, or a clear delayed-state message,
  within 5 minutes of purchase

SLI and SLO:
  recent-order visibility within 5 minutes
  target 99.9% over 30 days
  track receipt-delay support contacts as a product harm signal

Budget or release decision:
  pause non-urgent order-history releases during fast burn
  prioritize fixes if support contacts exceed agreed threshold

Degraded mode:
  do not show an empty list as if no order exists
  show "recent order still processing" with safe support path

Alerting or handoff:
  page only when recent-order visibility burns fast or receipt-delay support
  contacts spike; include dependency lag and affected order count

Cost/risk trade-off:
  payment correctness is safe, so 99.99 may not be worth buying yet
  invest first in lag SLI, delayed-state messaging, and support-safe lookup

Anti-pattern to avoid:
  metrics theater: API uptime does not prove users can see recent orders

A good answer does not need the same target number.

It does need a coherent chain:

promise -> SLI -> target -> decision -> degraded mode -> feedback -> trade-off

What Comes Next

The final capstone asks you to design the operating model yourself.

You will choose one service and define:

promise
SLIs
SLOs
error-budget policy
alerts
degraded modes
incident handoff
release safety
readiness evidence
control loop
cost/risk trade-off
anti-pattern review

The capstone is not a memory test.

It is a design and review task.

This synthesis lesson gives you the map. The capstone asks you to use it.

Resources

• [BOOK] Site Reliability Engineering: Service Level Objectives
  • Link: https://sre.google/sre-book/service-level-objectives/
  • Focus: Review how user promises become objectives and error-budget decisions.
• [BOOK] Site Reliability Workbook: Implementing SLOs
  • Link: https://sre.google/workbook/implementing-slos/
  • Focus: Use it for practical SLO design and stakeholder conversations.
• [BOOK] Site Reliability Workbook: Alerting on SLOs
  • Link: https://sre.google/workbook/alerting-on-slos/
  • Focus: Connect symptoms, budget burn, and human action.
• [BOOK] Site Reliability Workbook: Production Readiness Reviews
  • Link: https://sre.google/workbook/production-readiness/
  • Focus: Compare readiness artifacts with readiness evidence.
• [BOOK] Release It!: Design and Deploy Production-Ready Software
  • Link: https://pragprog.com/titles/mnee2/release-it-second-edition/
  • Focus: Revisit production failure patterns, stability mechanisms, and operational design.

Key Takeaways

• Reliability engineering starts with a user promise and becomes useful when that promise drives measurement, objectives, budgets, alerts, decisions, and learning.
• Internal metrics, alerts, SLOs, checklists, and automation are artifacts; they matter only when connected to an operating mechanism.
• A strong reliability review memo names the promise, SLI, SLO, budget decision, degraded mode, handoff, readiness evidence, control loop, economics, and anti-pattern risks.
• The main trade-off is not "reliability versus no reliability"; it is which reliability to buy, for which promise, at which cost, with which accepted risk.
• The final capstone will ask you to build this operating model for one service instead of recognizing the pieces in someone else's plan.