Reliability Reviews and Production Readiness

By the end of this lesson, you will be able to...

• Review whether a service is ready to make a reliability promise in production.

• Separate readiness evidence from checklist theater.

• Build a production-readiness review that connects promises, signals, alerts, degraded modes, incidents, and release decisions.

Idea in one sentence: Production readiness means the team has evidence that it can keep the promise, notice when it is failing, and act safely when production disagrees.

Core Insight

The checkout team wants to launch a new payment flow.

The feature looks good in staging.

Tests pass.

The happy path is smooth:

user clicks Pay
  -> checkout validates cart
  -> payment adapter authorizes payment
  -> order becomes paid
  -> receipt appears

Someone asks:

Are we ready for production?

The naive answer is:

Yes. The implementation is done and tests passed.

That answer is not enough.

Production readiness is not only about whether the code works in the best case. It is about whether the service can make a promise, measure the promise, operate the promise, and fail in ways the team can handle.

The previous lessons have given us the pieces:

• A user promise.
• SLIs and SLOs.
• Error budgets.
• Reliability dimensions and trade-offs.
• Toil and operational load.
• Symptom-based alerting.
• Capacity and saturation signals.
• Dependency degradation.
• Incident handoff.
• Release safety.

This lesson turns those pieces into a review.

Plain meaning:

Production readiness means the service is prepared to run under real users, real failures, real load, real dependencies, and real operating pressure.

In this scenario:

The new checkout payment flow is ready only if the team can explain what promise it makes, what evidence shows the promise is holding, what happens when payment is slow, and who acts when the promise is failing.

Technical name:

That review is a production-readiness review.

The Naive Checklist

Many readiness reviews become checklist theater.

Checklist theater sounds like this:

Monitoring? yes.
Alerts? yes.
Runbook? yes.
Tests? yes.
Owner? yes.
Rollback? yes.

Every box is checked.

Nobody can say whether the boxes prove anything.

This happens because a checklist can test possession instead of readiness.

Possession:

There is a dashboard.
There is an alert.
There is a runbook.
There is a rollback button.

Readiness:

The dashboard shows the user promise and the main failure modes.
The alert pages on urgent user impact and has a first action.
The runbook helps an on-call engineer make a safe first decision.
The rollback is safe for the current data and dependency state.

The difference is evidence.

A readiness review should not ask only:

Do we have this artifact?

It should ask:

What decision does this artifact help us make under pressure?
What evidence proves it works?

The Readiness Promise

Start with the promise.

For the new checkout payment flow:

Users should receive a clear checkout result within 2 minutes:
paid, declined, or safely pending.

Users should not be charged twice for one checkout attempt.

That promise is the anchor for the whole review.

Without it, readiness becomes vague.

People argue about whether the service is "stable," "observable," or "safe" without naming what safe means.

With the promise, the review can ask concrete questions:

Can we measure clear result within 2 minutes?
Can we detect duplicate-charge risk?
Can we preserve pending state when payment is unknown?
Can we explain which users are affected?
Can we decide when to pause releases?

A readiness review should reject promises that are too broad.

Bad promise:

Checkout should be reliable.

Better promise:

Checkout should give users a truthful final or pending state within 2 minutes
and should not create duplicate charges for one checkout attempt.

The better promise can be measured, reviewed, and operated.

Check: Why is "checkout should be reliable" weak as a production-readiness promise?

Think first, then reveal.

Answer: It does not say what user outcome must hold, what time window matters, or what failure must be avoided. A readiness review needs a promise specific enough to measure and operate.

A Worked Readiness Review

Review the new payment flow.

Use a table, but make every row evidence-based.

Now trace one review item more deeply.

Input:

The review asks:
  "What happens when payment provider p95 latency rises above 8 seconds?"

Naive answer:

The provider is slow, so checkout will retry.

Better review path:

1. User-path timeout is bounded at 2 seconds.
2. Checkout writes `payment_pending` with idempotency key.
3. User sees a truthful pending message.
4. Reconciler retries later with caps.
5. Queue age and duplicate-prevention hits are monitored.
6. Clear-result SLO fast burn pages if the promise is failing.
7. Support receives wording for pending-payment questions.
8. Release owner pauses risky checkout releases while budget burns.

Intermediate state:

The provider is still unhealthy.
Checkout is not pretending success.
Users get a clear state.
The team has signals and owners.

Output:

The service keeps a smaller promise instead of turning one dependency failure
into latency, retries, duplicate risk, and support confusion.

Naive failure contrast:

If the review only asks whether retries exist, it misses the real risk.
Retries without bounded waiting, pending state, idempotency, and queue-age
signals can make the outage worse.

That is the point of a readiness review.

It makes failure paths visible before production forces the lesson.

Production Readiness Is Evidence, Not Perfection

Readiness does not mean the service cannot fail.

A service that cannot fail is not a real production service.

Readiness means the team can answer:

What promise are we making?
How will we know when it is failing?
What will we do first?
What is the safe degraded behavior?
Who owns the next decision?
What risk are we choosing to accept?

Some answers may be imperfect.

That is allowed if the risk is explicit.

For example:

Known gap:
  support does not yet have a self-service view for pending payment state

Risk:
  during provider outage, engineering may receive repeated support questions

Temporary control:
  on-call can export affected order IDs for support every 15 minutes

Follow-up:
  build support-safe lookup before increasing checkout traffic by 3x

This is not ideal.

But it is honest.

The dangerous version is:

Support is fine.

with no evidence and no control.

Production readiness can accept risk. It should not hide risk.

The Review Meeting

A good readiness review is small and concrete.

It does not need to become a courtroom.

For the checkout payment flow, invite people who own decisions:

service owner
on-call representative
release owner
support representative
product owner when user-visible degradation matters
dependency owner if the dependency is critical

The meeting should produce decisions, not just discussion.

Useful outputs:

approved for production
approved with constraints
not ready until specific blockers are fixed
accepted risk with owner and review date
launch limit, such as region, traffic percent, or customer segment

Examples:

Approved with constraints:
  launch to 10% of checkout traffic
  do not launch during active payment-provider incident
  keep promotion traffic disabled for first 24 hours
  on-call watches clear-result SLI and pending queue age
  support gets pending-payment wording before launch

Not ready:
  no duplicate-charge risk signal
  rollback cannot read new pending-payment state
  no support path for ambiguous payment state

The first outcome is a controlled launch.

The second outcome is not bureaucracy. It is the review doing its job.

Check: A service has dashboards and alerts, but nobody can say what user promise they protect. Is that production-ready evidence?

Think first, then reveal.

Answer: No. Dashboards and alerts are only useful readiness evidence when they connect to a promise, a decision, and an action.

Blockers Versus Accepted Risk

A readiness review should separate two very different outcomes:

This is risky, but we accept it for now.
This is risky enough that launch should stop.

Accepted risk has an owner, a reason, and a review point.

Example:

Risk:
  support lookup for pending payments is manual for the first week

Why accepted:
  launch is limited to 5% of traffic
  support volume is expected to be low
  on-call can export affected order IDs during business hours

Owner:
  checkout service owner

Review:
  revisit before moving above 25% traffic

A blocker is different.

Example:

Blocker:
  old code cannot read the new pending-payment state

Why it blocks:
  rollback may strand ambiguous payment attempts
  duplicate-charge protection may become harder during recovery

Required before launch:
  make old and new code read both states
  test rollback with pending-payment records

The difference is not whether someone is worried. The difference is whether the team has a safe operating path if the risk becomes real.

If there is a safe temporary control, the review can accept the risk with constraints.

If there is no safe control, the review should block the launch.

This distinction keeps the review practical. It avoids two bad habits: blocking everything because risk exists, or launching everything because the risk is merely written down.

The best reviews also change the design before launch.

Suppose the team notices that the pending-payment degraded mode exists, but support cannot see whether a user's order is pending, paid, declined, or stuck. That is not only a support gap. It changes the reliability behavior of the service. Users will ask for help, support will escalate to engineering, on-call will search logs, and the same manual lookup will repeat during every provider incident.

That is the same pattern we called toil earlier in the track.

The review can turn that discovery into a design change:

• Add a support-safe payment-state lookup.
• Include pending-state age and last reconciliation attempt.
• Show whether the user should retry, wait, or contact support.
• Record enough evidence without exposing sensitive payment details.

Now the service is more ready, not because the checklist is prettier, but because one failure path has a clearer operating surface.

This is the spirit of production readiness. The review should make the service easier to operate under stress. If it only produces a document that nobody uses during an alert, it has missed the point.

Trade-offs and Limits

Readiness reviews improve reliability because they force hidden assumptions into the open before users depend on them.

They help teams find missing signals, unsafe rollback paths, fake degraded modes, unclear ownership, and expected toil.

They cost time and attention.

If every tiny change requires a heavy review, teams will avoid the process or treat it as paperwork. The review should match the risk of the service and change.

It does not replace runtime learning.

Production will still reveal surprises. A readiness review is not a guarantee. It is a structured bet: "Given what we know, this is the promise, evidence, risk, and operating plan."

You can see the boundary when:

the review checks boxes but does not change any decision
the same concerns return every launch without owners
teams write long documents no on-call engineer can use
the service passes review but has no signal for its core promise

The trade-off is:

Too little review ships hidden risk.
Too much generic review creates ceremony.
Good readiness review is specific, evidence-based, and tied to launch decisions.

Common Confusions

Confusion: "Production readiness means all risks are gone"

Why it is tempting:

The word ready sounds final.

Better model:

Ready means the important risks are named, measured where possible, assigned to owners, and accepted or blocked deliberately.

Confusion: "A checklist is the review"

Why it is tempting:

Checklists are easy to reuse and easy to audit.

Better model:

A checklist is a prompt. The review is the evidence and decision behind each item.

Confusion: "Staging success proves production readiness"

Why it is tempting:

Staging feels safer and controlled.

Better model:

Staging can catch many defects, but production readiness also depends on real traffic, dependency behavior, on-call response, support flow, capacity margin, and release reversibility.

Confusion: "Readiness belongs only before launch"

Why it is tempting:

Teams often review a service once, then move on.

Better model:

Readiness changes when traffic, dependencies, architecture, team ownership, or user promises change. A service can become less ready over time.

Practice

Review this readiness claim:

The new checkout payment flow is ready.
It passed tests.
It has a dashboard.
It has an alert for provider timeout rate.
Rollback exists.

Rewrite it as a real readiness review.

Include:

1. User promise.
2. SLI/SLO evidence.
3. Degraded mode.
4. Alerting and incident handoff.
5. Release safety.
6. Launch decision.

Model answer:

User promise:
  users receive paid, declined, or safely pending within 2 minutes;
  no duplicate charge for one checkout attempt

SLI/SLO:
  clear-result ratio measured on all checkout attempts
  duplicate-prevention hit rate monitored
  99.5% clear-result SLO with fast-burn alert

Degraded mode:
  provider timeout creates payment_pending with idempotency key
  user receives honest pending message
  reconciler has retry caps and support-review state

Alerting and handoff:
  page on clear-result fast burn or duplicate-risk signal
  handoff packet includes impact, timeline, mitigation, unknowns, owners

Release safety:
  rollback can read pending-payment state
  launch pauses if SLO burn starts or queue age exceeds degraded-mode target
  no unrelated checkout releases during active burn

Decision:
  approved with constraints:
  launch to limited traffic, watch SLI and queue age, give support pending
  wording before launch, review after 24 hours

Resources

• [BOOK] Site Reliability Workbook: Production Readiness Reviews
  • Link: https://sre.google/workbook/production-readiness/
  • Focus: Use it for the structure of readiness evidence and operational review.
• [BOOK] Site Reliability Engineering: Service Level Objectives
  • Link: https://sre.google/sre-book/service-level-objectives/
  • Focus: Connect readiness to explicit service promises and measurable objectives.
• [BOOK] Site Reliability Engineering: Monitoring Distributed Systems
  • Link: https://sre.google/sre-book/monitoring-distributed-systems/
  • Focus: Review why monitoring should support action, not just visibility.
• [BOOK] Release It!: Design and Deploy Production-Ready Software
  • Link: https://pragprog.com/titles/mnee2/release-it-second-edition/
  • Focus: Read for practical production failure patterns and readiness thinking.

Key Takeaways

• Production readiness is evidence that the team can make, measure, operate, and safely degrade a reliability promise.
• A checklist is useful only when each item connects to evidence and a decision.
• Readiness reviews should expose missing signals, unsafe degraded modes, unclear ownership, expected toil, and risky rollback paths.
• A service can be approved with constraints, blocked until specific risks are fixed, or launched with explicit accepted risk.
• Readiness is not permanent; it changes when traffic, dependencies, architecture, ownership, or the user promise changes.