Reliability Capstone: Define and Operate a Service Promise

By the end of this lesson, you will be able to...

• Design a reliability operating model for one backend service.

• Connect a user promise to SLIs, SLOs, error-budget policy, alerts, degraded modes, readiness evidence, and control loops.

• Review your own reliability plan for trade-offs, accepted risk, and anti-patterns.

Idea in one sentence: A reliable service is not one with many reliability artifacts; it is one whose user promise drives measurement, decisions, action, and learning.

Core Insight

This capstone asks you to do the thing the whole track has been building toward.

You will define and operate a reliability promise for one service.

That means more than writing:

The service should be reliable.

It also means more than writing:

SLO: 99.9%

A real reliability operating model answers a chain of questions:

What user outcome are we promising?
How do we measure that outcome?
What level of imperfection do we accept?
What happens when the budget burns?
What smaller promise can we keep during failure?
Who acts, using which signal, and with which first action?
What evidence proves the service is ready?
What feedback tells us whether our action helped?
What cost and risk did we choose?
What anti-patterns are we avoiding?

The capstone is not a memory test.

It is a design review.

Your job is to produce a short reliability plan that a service owner, on-call engineer, release owner, and product partner could actually use.

The plan should be concrete enough to make decisions under pressure.

Think of the plan as a small operating contract.

It should tell a tired on-call engineer what matters first. It should tell a release owner when the service can no longer afford extra change risk. It should tell a product partner which user harm the team is trying to prevent, and which risk the team is accepting for now.

That is why the capstone asks for both design and operation. A design that names a beautiful promise but has no alert, degraded mode, or feedback loop will not help during production pressure. An operations plan that has alerts and runbooks but no clear user promise will create noise. The strong answer joins both sides.

The Scenario

Use this service for the capstone:

Service:
  order history

What it does:
  shows users their recent orders, receipts, and order state

Dependencies:
  order database
  receipt generation worker
  payment event stream
  support lookup tool

Known behavior:
  payment correctness is handled by the payment system
  order history sometimes lags when the receipt worker or event stream is slow
  users contact support when a recent order disappears or receipt is delayed
  support often escalates to engineering because it cannot see enough state

Product context:
  delayed receipts are painful and reduce trust
  they are less severe than duplicate charges
  enterprise customers care about auditability
  the product team wants faster checkout experiments next quarter

You may imagine normal traffic for the service.

Keep the design small enough to inspect.

Do not solve observability platform design, incident command structure, chaos testing, or capacity forecasting in full depth. Those belong to downstream tracks.

This capstone stays inside the foundation promise:

define a service promise
measure it
operate it
make trade-offs explicit

Constraints

Your plan must respect these constraints:

1. Payment correctness is not owned by order history.
2. Order history can show honest delayed state.
3. Support needs enough information to answer users safely.
4. Engineering time is limited.
5. The team cannot buy extreme reliability for every path.
6. Release speed matters, but not during active reliability burn.
7. The service depends on asynchronous events, so "latest state" can lag.

These constraints matter.

They prevent the easy but weak answer:

Make everything 99.99% and alert on all failures.

The better answer chooses which promise deserves which protection.

This is also a boundary exercise.

Order history depends on payment events, receipt workers, databases, and support tooling. It can make some promises directly, and it must be careful with others. If order history claims to guarantee payment correctness, it is promising something owned by another service. If it refuses to say anything about recent order visibility, users are left with a vague product experience.

Good reliability design lives between those mistakes. It says what this service can own, what it can observe, what it can do during failure, and when it must hand off to another owner.

Design Goal

Create a reliability operating model with these sections:

1. User promise
2. SLIs
3. SLO and error-budget policy
4. Reliability dimensions and trade-offs
5. Degraded mode
6. Alerting and handoff
7. Release safety
8. Production readiness evidence
9. Control loop
10. Reliability economics
11. Anti-pattern review

The plan should fit on one or two pages.

It should be specific, not encyclopedic.

The standard is:

Could someone use this plan to make a safer decision during a real reliability problem?

Use that standard while writing.

For every section, ask whether the sentence would change behavior. If it would not change measurement, alerting, release decisions, support behavior, degraded mode, or product risk, it may be decorative. Decorative text can still sound professional, but it will not help under pressure.

For example, "monitor event lag" is weaker than "page when recent-order visibility burns fast; use event lag to decide whether to involve the event-stream owner." The second sentence connects evidence to ownership and action. That is the level of specificity this capstone is looking for.

Proposed Model

Here is a strong model answer.

Do not memorize it.

Use it as a reference for the shape and level of detail your own answer should have.

1. User Promise

Users can see recent orders and receipts within 5 minutes of purchase,
or they receive a clear delayed-state message.

Support can inspect the order state safely when a user asks for help.

This promise is better than:

Order history should be reliable.

The better promise names:

user outcome:
  recent orders and receipts are visible

time window:
  within 5 minutes

honest degraded behavior:
  clear delayed-state message

operational support:
  support can inspect state safely

It also avoids claiming ownership of payment correctness.

That boundary matters.

Order history should not promise:

Payment was definitely captured correctly.

It can promise:

We show the best known order and receipt state clearly.

This is a small but important reliability habit.

A promise should be ambitious enough to protect the user, but honest enough that the service can operate it. If the promise is too broad, the team will not know what to measure or who owns failure. If the promise is too narrow, users can still be harmed while the service looks healthy on paper.

The order-history promise is useful because it names the user-visible state and the time window. It also says what happens when the ideal state is not available. That "or" clause matters: "or they receive a clear delayed-state message" turns a failure into a smaller promise the team can still keep.

2. SLIs

Choose indicators that measure the promise.

Primary SLI:
  recent_order_visibility_ratio
  percentage of completed purchases where the order appears in order history
  within 5 minutes

Secondary SLI:
  receipt_visibility_ratio
  percentage of completed purchases where receipt is visible within 5 minutes

Supporting signals:
  receipt worker queue age
  payment event stream lag
  order database read latency
  support contacts tagged "missing recent order"
  support contacts tagged "missing receipt"

The primary SLI measures the user promise.

The supporting signals explain pressure.

Do not confuse them.

CPU and memory may help debug the service, but they do not prove that users can see recent orders.

When choosing SLIs, prefer the signal closest to the user's experience that the service can measure honestly.

That does not mean supporting metrics are unimportant. Event lag, worker queue age, and database read latency may explain why users cannot see receipts. They help with diagnosis and control-loop decisions. But they should not replace the outcome metric.

This is the mistake behind many reliability plans. They measure what is convenient, then accidentally optimize for it. The capstone should show that you can separate the outcome signal from the explanatory signals.

3. SLO and Error-Budget Policy

SLO:
  99.9% of completed purchases show recent order state within 5 minutes,
  measured over 30 days.

Receipt objective:
  99.5% of receipts visible within 5 minutes,
  measured over 30 days.

Budget policy:
  if recent_order_visibility_ratio burns fast,
  pause non-urgent order-history releases.

  if receipt visibility burns but order state is correct,
  route first to degraded messaging and support workflow,
  then prioritize receipt worker fixes based on support impact.

  if support contacts exceed the agreed threshold,
  product and engineering review whether the target is too loose
  or the degraded message is unclear.

This policy separates order state from receipt display.

That is important.

If order state is visible but receipt generation lags, the user harm is real but not the same as payment uncertainty.

The policy should reflect that difference.

Notice that this section turns the SLO into behavior.

The target is not just a number. It changes release policy, prioritization, support review, and product discussion. If the budget burns and nothing changes, the SLO is decoration. If every tiny burn creates the same emergency, the SLO may be too strict or the policy too blunt.

A good budget policy has different responses for different kinds of harm. Missing order state, delayed receipt display, and support confusion should not all receive the same response if their user harm and recovery paths differ.

4. Reliability Dimensions and Trade-offs

Order history has several reliability dimensions:

Availability:
  can users open order history?

Latency:
  does the page respond quickly?

Freshness:
  does recent order state appear within the promised window?

Correctness:
  does the page avoid claiming false order or payment state?

Supportability:
  can support answer a user without asking engineering to search logs?

The central trade-off:

Showing an honest delayed state may feel less polished than showing the final receipt,
but it is safer than pretending the order does not exist.

A second trade-off:

Strict freshness targets protect user trust,
but they may require more worker capacity, better event-lag monitoring,
support tooling, and slower releases during budget burn.

The capstone should name trade-offs without treating them as excuses.

Saying "this is expensive" is not enough. The plan should say what the expense buys and what risk remains. The team might decide that better support tooling is worth buying before multi-region receipt generation. That is not lowering the reliability bar. It is choosing the next investment that best reduces current user harm.

Strong trade-off writing has a shape: this improves one thing, costs another thing, and leaves a named residual risk. That shape keeps reliability decisions honest.

5. Degraded Mode

The naive degraded mode is:

If the order is not ready, show an empty order list.

That is dangerous.

The user may think the purchase failed.

A better degraded mode:

If a completed purchase is known but receipt is delayed:
  show "Your order is still processing. This usually updates within a few minutes."

If order state is delayed beyond 5 minutes:
  show a delayed-state message with support-safe reference ID.

If support opens the lookup tool:
  show last known order state, receipt worker state, event lag,
  and whether payment correctness is owned by another service.

This degraded mode keeps a smaller truthful promise.

It does not hide uncertainty.

It gives the user and support a safer next step.

Degraded modes are not only technical fallbacks.

They are user communication decisions. An empty order list is technically easy, but it tells the wrong story. It suggests that no order exists. A delayed-state message tells the user that the system has not forgotten them, while also avoiding a false claim that the receipt is ready.

This is why degraded modes belong in reliability foundations. They connect system truth to user trust. A service can be less complete for a short period and still be more reliable than a service that pretends everything is fine.

6. Alerting and Handoff

Alert on user impact and actionable response.

Page:
  recent_order_visibility_ratio fast burn
  order state missing for more than 5 minutes for a high number of users
  support-contact spike for missing orders

Do not page by itself:
  CPU above 70%
  one worker restart
  raw request count increase

The first alert action:

1. Check event stream lag and receipt worker queue age.
2. Confirm whether order database reads are healthy.
3. Enable delayed-state messaging if not already active.
4. Pause non-urgent order-history releases.
5. Send support an affected-order count and current user wording.

Incident handoff should include:

impact:
  how many users cannot see recent orders or receipts

timeline:
  when lag started and what changed recently

current mitigation:
  delayed-state messaging, release pause, worker scaling, or queue drain

unknowns:
  whether event lag, worker failure, database read latency, or deploy caused it

owners:
  order-history on-call, receipt-worker owner, support contact, release owner

next review:
  when to check queue age and SLI again

The alert and handoff sections should make ownership visible.

An alert without a first action asks the on-call engineer to invent the plan while the service is already under pressure. A handoff without impact, timeline, mitigation, unknowns, and next owner makes the next team rediscover the same facts.

Good reliability work reduces repeated thinking during stress. It does not remove judgment, but it gives judgment a better starting point.

7. Release Safety

Order history releases can continue when budget is healthy.

During fast burn, the policy changes:

Pause:
  non-urgent UI changes
  schema changes affecting order state reads
  worker changes unrelated to mitigation

Allow:
  targeted fixes that reduce current impact
  observability changes that improve the current decision
  support wording changes that reduce confusion

Require rollback check:
  old code can read any new delayed-state field
  support tool remains compatible with both versions

This is not "stop all change forever."

It is choosing change risk based on current reliability evidence.

Release safety is often where reliability becomes practical.

Many teams agree with reliability in principle, then keep shipping risky changes during active burn because there is no policy that connects the budget to release decisions. The capstone should avoid that gap. If the service is already failing its promise, the plan must say which changes pause, which changes continue, and who can approve exceptions.

The best policy is not the strictest possible policy. It is the policy that reduces avoidable risk while still allowing targeted recovery work.

8. Production Readiness Evidence

Before increasing order-history traffic or launching a new receipt flow, require evidence:

Promise:
  written user promise and known non-promises

SLI:
  recent-order visibility measured from completed purchases

SLO:
  99.9% target and error-budget policy agreed by engineering and product

Degraded mode:
  delayed-state message tested

Support:
  support-safe lookup tested with delayed and completed states

Rollback:
  old and new code read delayed-state fields safely

Alert:
  fast-burn alert includes first action and escalation condition

Load:
  worker queue age and event lag observed under expected traffic

Handoff:
  incident packet template exists and has owners

A checkbox is not enough.

Each item needs evidence that it can change a decision.

Readiness evidence should be testable.

"Rollback exists" is weak. "Old and new code can both read delayed-state fields, and rollback was tested with delayed receipt records" is much stronger. "Support has a tool" is weak. "Support can see last known order state, receipt worker state, and safe user wording" is stronger.

This is the difference between possession and readiness. Possession means the artifact exists. Readiness means the artifact can be used under the failure path the service is likely to face.

9. Control Loop

Define the loop before the incident.

Target:
  recent orders visible within 5 minutes

Sensor:
  recent_order_visibility_ratio
  receipt_visibility_ratio
  worker queue age
  event stream lag
  support contacts

Comparator:
  fast SLO burn
  queue age above degraded-mode threshold
  support contacts above product threshold

Controller:
  order-history on-call and release owner

Control surface:
  enable delayed-state messaging
  pause releases
  scale receipt workers within safe limits
  cap client retries
  route support workflow

Feedback delay:
  10 minutes for queue-age direction
  15 minutes for recent-order visibility trend
  longer for support-contact reduction

Learning:
  if queue age falls but support contacts remain high,
  improve user wording or support tooling before only adding capacity

The loop prevents random action.

It tells the team what to watch after changing the system.

The feedback delay is especially important.

Without it, teams often stack actions too quickly. They scale workers, roll back a deploy, change retries, and update messaging all at once, then cannot tell which action helped. The control loop makes the next observation explicit. It gives the system enough time to respond before the team changes another variable.

That does not mean waiting forever. It means choosing a delay that matches the mechanism: queue age may move in minutes, support contacts may take longer, and product trust may take longer still.

10. Reliability Economics

Do not buy the maximum target immediately.

A reasonable first decision:

Buy:
  recent-order visibility SLI
  99.9% recent-order visibility SLO
  delayed-state messaging
  support-safe lookup
  receipt worker queue-age alerting
  release pause during fast burn

Do not buy yet:
  multi-region receipt generation
  99.99% receipt visibility
  24/7 dedicated order-history incident commander

Accepted risk:

Rare receipt delays may still happen.
They are acceptable if order state is truthful,
users see clear delayed-state messaging,
support can inspect state,
and support-contact volume stays below the threshold.

Review trigger:

Revisit the target if:
  enterprise customers require stricter receipt auditability
  delayed receipts cause measurable churn
  support contacts remain high after tooling and messaging
  budget burn repeatedly pauses important product work

This is the cost/risk decision.

It says what the team buys now, what it does not buy yet, and what evidence would change the decision.

This section should feel like a real product-engineering conversation.

It is not enough to say "we accept the risk." Accepted risk needs a reason, an owner, and a review trigger. It is also not enough to say "we need maximum reliability." Maximum reliability everywhere can consume attention that should protect higher-harm promises.

Reliability economics asks for proportionality. The plan should spend reliability effort where it most protects user trust, safety, correctness, business value, or operational sustainability.

11. Anti-pattern Review

Check the plan against common traps.

Metrics theater:
  Avoided because primary SLI measures recent-order visibility,
  not only CPU or request count.

SLO decoration:
  Avoided because budget burn changes release policy and prioritization.

Noisy paging:
  Avoided because pages require user impact and first action.

Checklist theater:
  Avoided because readiness items require evidence.

Pattern copying:
  Avoided because retries and worker scaling have caps and feedback delay.

Heroics:
  Reduced because support lookup and handoff packet replace repeated manual log searches.

This is the final pass.

It checks whether the plan is an operating model, not a pile of artifacts.

Anti-pattern review is useful because capstone answers can look mature while still being hollow.

A plan with many metrics may still miss the user outcome. A plan with a strict SLO may still have no budget policy. A plan with a readiness checklist may still lack evidence. A plan with automation may still have no feedback delay.

The anti-pattern pass is the final guardrail. It asks whether each artifact has a job in the operating model.

Walkthrough

Now run one incident through the model.

Starting state:

09:50  new receipt-worker version deployed
10:05  event stream lag normal
10:10  receipt worker queue age rises to 8 minutes
10:15  recent_order_visibility_ratio starts burning fast
10:18  support contacts for missing receipts spike

Naive response:

Restart the API service and increase client retries.

Why that is weak:

API uptime is not the main user promise.
Client retries add more load but do not fix delayed receipt generation.
Support still cannot answer users.

Better response:

1. Confirm recent-order visibility burn and receipt queue age.
2. Pause non-urgent order-history releases.
3. Enable delayed-state messaging.
4. Check whether receipt-worker deploy is compatible with rollback.
5. Roll back receipt worker if evidence points to the deploy and rollback is safe.
6. Send support affected count, current state, and user wording.
7. Recheck queue age after 10 minutes and recent-order visibility after 15 minutes.

Intermediate state:

Users see an honest delayed state.
Support can answer what is known.
Release risk is reduced.
The team is watching the signal that matches the promise.

Decision after feedback:

If queue age falls:
  keep delayed-state messaging until visibility SLI returns to normal
  review why support contacts still rose

If queue age rises:
  escalate to receipt-worker owner
  consider worker rollback or safe capacity increase
  keep release pause active

What we learned:

The plan turns one vague symptom into user impact, signal, decision,
action, feedback, and learning.

That is the capstone standard.

The walkthrough is where weak plans usually reveal themselves.

If you cannot run a realistic incident through the plan, the plan is probably too abstract. The walkthrough should show what changes first, who acts, what is observed, what action is taken, what the team waits for, and what decision happens next.

This is also where trade-offs become visible. Enabling delayed-state messaging may reduce user confusion but increase visible degraded responses. Scaling workers may drain a queue but create database pressure. Rolling back may help only if compatibility is safe. A strong capstone does not hide these tensions.

Failure Review

Your plan should be able to name what can still fail.

For order history:

The SLI may miss users who refresh many times.
Support tags may be incomplete.
Delayed-state wording may reduce confusion but not eliminate it.
Worker scaling may increase database pressure.
Rollback may be unsafe after a schema change.
Enterprise customers may need stricter auditability than ordinary users.

Do not hide these limits.

Named limits are part of a mature plan.

The plan is weaker if it claims:

This solves order-history reliability.

The plan is stronger if it says:

This protects recent-order visibility and supportability.
It does not guarantee instant receipt generation.
It does not own payment correctness.
It must be reviewed if user harm changes.

Failure review is not pessimism.

It is how the plan avoids false confidence. A reliability plan that says what it does not solve is more trustworthy than one that claims complete protection. It also gives future teams the right next questions when conditions change.

For example, enterprise auditability might become more important later. If that happens, the current target may no longer be enough. The capstone should make that kind of review trigger explicit.

Final Challenge

Create your own reliability operating model for one service.

Use the order-history scenario above, or choose a similar backend service you understand:

checkout payment flow
order history
notification delivery
search API
file upload
account login
report generation

Your final answer should include:

1. User promise.
2. Non-promises or ownership boundaries.
3. Primary SLI and supporting signals.
4. SLO and error-budget policy.
5. Reliability dimensions and trade-offs.
6. Degraded mode.
7. Alerting and first action.
8. Incident handoff evidence.
9. Release safety policy.
10. Production readiness evidence.
11. Control loop with feedback delay.
12. Reliability economics and accepted risk.
13. Anti-pattern review.

Keep it concrete.

Do not write:

Monitor everything and alert the team.

Write:

Page on recent-order visibility fast burn.
First action: enable delayed-state messaging, check worker queue age,
pause non-urgent releases, and send support an affected-order count.

The second version can be used.

The first version cannot.

Write your final answer as if another engineer will inherit it.

That engineer should not need to guess what "monitor closely" means. They should know which signal matters, which threshold changes the decision, what first action is safe, what feedback delay to expect, and who receives the handoff if the first action does not work.

The capstone is successful when the plan reduces ambiguity without pretending that production is perfectly predictable.

Rubric

Use this rubric to evaluate your capstone.

The capstone passes when the plan can guide a real decision.

You can use the rubric in two passes.

First, check completeness: did you include every required part? Second, check mechanism: does each part influence measurement, decision, action, learning, or trade-off? Completeness without mechanism is checklist theater. Mechanism without completeness may leave important failure paths uncovered.

If one row is weak, revise the plan before adding more sections. A sharper promise, a better SLI, or a clearer degraded mode often improves the whole plan more than extra pages of generic operational text.

Resources

• [BOOK] Site Reliability Engineering: Service Level Objectives
  • Link: https://sre.google/sre-book/service-level-objectives/
  • Focus: Use it to check whether your promise, SLI, SLO, and budget connect cleanly.
• [BOOK] Site Reliability Engineering: Embracing Risk
  • Link: https://sre.google/sre-book/embracing-risk/
  • Focus: Use it to frame accepted risk and reliability economics.
• [BOOK] Site Reliability Workbook: Alerting on SLOs
  • Link: https://sre.google/workbook/alerting-on-slos/
  • Focus: Use it to review whether your alerts page on user impact and action.
• [BOOK] Site Reliability Workbook: Production Readiness Reviews
  • Link: https://sre.google/workbook/production-readiness/
  • Focus: Use it to turn checklist items into readiness evidence.
• [BOOK] Release It!: Design and Deploy Production-Ready Software
  • Link: https://pragprog.com/titles/mnee2/release-it-second-edition/
  • Focus: Use it to think through production failure modes, boundaries, and operational design.

Key Takeaways

• The capstone is a reliability operating model, not a list of reliability tools.
• A strong plan starts with a user promise and keeps that promise connected to SLIs, SLOs, budgets, alerts, degraded modes, readiness, control loops, economics, and anti-pattern review.
• Ownership boundaries matter because a service should not promise outcomes it cannot control.
• The most useful reliability plans name both action and feedback delay, so teams can learn whether an action helped.
• A plan is ready when it can guide a safer decision during real production pressure.