Reliability Anti-Patterns and Cargo Cults

By the end of this lesson, you will be able to...

• Recognize reliability practices that look mature but do not protect a user promise.

• Diagnose whether a metric, alert, checklist, SLO, or automation has the mechanism it needs.

• Rewrite a shallow reliability practice into one connected to evidence, decision, action, and learning.

Idea in one sentence: A reliability practice is useful only when it connects a user promise to evidence, ownership, action, and a cost-aware decision.

Core Insight

The checkout team has learned a lot.

It can now talk about:

user promises
SLIs and SLOs
error budgets
availability, latency, durability, and correctness
toil
symptom-based alerting
capacity and saturation
graceful degradation
incident handoff
release safety
production readiness
control loops
reliability economics

That is a useful toolkit.

It also creates a new risk.

A team can copy the visible shape of mature reliability work without copying the mechanism that makes it useful.

It can have:

dashboards
alerts
SLOs
readiness reviews
runbooks
post-incident actions
automation
error-budget policy

and still not be able to answer:

What user promise is protected?
What signal proves the promise is failing?
Who decides what happens next?
What action changes the system?
What cost or risk are we accepting?
What did we learn after the action?

That gap is the heart of reliability anti-patterns.

Plain meaning:

An anti-pattern is a practice that looks reasonable, and may even be common, but repeatedly creates bad results because the underlying mechanism is wrong or missing.

In this scenario:

A checkout dashboard full of CPU, pod restarts, and provider timeout charts may look operationally mature. But if it does not show whether users receive paid, declined, or safely pending within 2 minutes, it does not protect the checkout promise.

Technical name:

When a team copies the visible ritual without the working mechanism, people often call it a cargo cult practice.

The goal of this lesson is not to mock teams.

Most anti-patterns start from a reasonable need. A team wants to be safer, faster, more mature, or more accountable.

The review skill is to ask:

What is this practice trying to protect?
What mechanism makes it work?
What evidence says it is working?

The Diagnostic Question

Use one question for every reliability practice:

What decision does this help us make?

Then ask the follow-up questions:

What user promise is involved?
What signal is used?
What threshold or comparison matters?
Who owns the decision?
What action is available?
What trade-off does the action introduce?
What feedback tells us whether it helped?

If a practice cannot answer those questions, it may be decorative.

Decorative does not mean useless forever.

It means the practice is not yet connected to the reliability operating model.

For example:

Dashboard:
  maybe useful as raw evidence
  not useful as a paging signal unless it maps to user impact

Checklist:
  maybe useful as a prompt
  not useful as readiness evidence unless each item changes a launch decision

Automation:
  maybe useful as a fast actuator
  not useful unless the signal, safety limit, and feedback delay are understood

This is the same model from the previous lessons.

Reliability is not the artifact.

Reliability is the promise, signal, decision, action, learning loop, and cost-aware trade-off behind the artifact.

Check: A team says, "We have an SLO, so reliability is covered." What is the first diagnostic question?

Think first, then reveal.

Answer: Ask what decision the SLO drives. If the SLO does not affect alerting, release safety, prioritization, accepted risk, or product trade-offs, it may be only a decorative target.

Anti-Pattern 1: Metrics Theater

Metrics theater happens when a team measures many things but still cannot see the user promise.

It often looks impressive:

CPU usage
memory usage
pod restarts
request count
database connections
provider timeout rate
queue length
deployment frequency

Those metrics can be useful.

The anti-pattern appears when they replace the user-centered SLI.

For checkout, the promise is not:

CPU stays below 70%.

The promise is:

Users receive paid, declined, or safely pending within 2 minutes.
Users are not charged twice for one checkout attempt.

CPU may help explain why the promise is failing.

It does not prove the promise is holding.

Repair:

Start with the promise.
Choose one or two user-centered SLIs.
Keep internal metrics as supporting evidence.
Do not page on supporting evidence unless it predicts or explains user impact.

The useful distinction is:

SLI:
  measures whether the promise is holding

supporting metric:
  helps explain why the promise may be failing

Metrics theater confuses the second with the first.

Anti-Pattern 2: Noisy Paging As Responsibility

Noisy paging feels like seriousness.

The team can say:

We page quickly.
We do not ignore failures.
Someone is always watching.

But paging is only useful when a human can make a better decision now.

An alert is weak when it says:

provider timeout rate is high

and the on-call engineer has no clear first action, no user-impact context, and no threshold for escalation.

The stronger alert says:

checkout clear-result SLO is burning fast
pending queue age is above degraded-mode target
first action: keep provider circuit open, cap retries, pause promotion
escalate if queue age keeps rising after 15 minutes

The anti-pattern is not "too many alerts" only.

The anti-pattern is paging without an action.

Repair:

Page on urgent user impact.
Include the first safe action.
Include the evidence needed for handoff.
Route non-urgent signals to review, not wake-ups.

The trade-off is real:

Fewer pages can miss urgent failures if the signal is too narrow.
More pages can train humans to ignore noise.
Good paging chooses signals where human action now can reduce user harm.

Anti-Pattern 3: SLO Decoration

SLO decoration happens when the team writes a target but does not use it.

It looks like this:

Checkout SLO:
  99.9% clear result within 2 minutes

Reality:
  releases continue during fast burn
  alerting pages on provider timeout only
  product does not know what risk is accepted
  support does not have a pending-payment workflow
  retros do not revisit the target

The SLO exists.

The operating model does not.

A real SLO should influence decisions:

When do we page?
When do we pause change?
When do we spend engineering time on reliability?
When do we accept risk?
When do we revisit the target because product harm changed?

Repair:

Attach the SLO to an error-budget policy.
Name what happens during fast burn.
Name what happens when budget is healthy.
Review whether the target still matches user harm and cost.

An SLO is not a trophy.

It is a decision boundary.

Anti-Pattern 4: Checklist Theater

Checklist theater is a readiness review that checks possession instead of evidence.

It sounds like this:

Dashboard? yes.
Alert? yes.
Runbook? yes.
Owner? yes.
Rollback? yes.

The list is not wrong.

The missing part is proof that each item supports an operating decision.

Better readiness questions:

Dashboard:
  Which user promise does it show?

Alert:
  What first action does it trigger?

Runbook:
  Can a tired on-call engineer use it in the first 10 minutes?

Owner:
  Which decision does this person own?

Rollback:
  Can old code read the new state safely?

Repair:

Keep the checklist as a prompt.
Require evidence for each item.
Record launch decision, accepted risk, owner, and review trigger.

The trade-off:

No checklist lets teams forget important failure paths.
Only checklist creates ceremony.
Evidence-based readiness sits between those failures.

Anti-Pattern 5: Pattern Copying Without System Boundaries

Reliability patterns are useful.

Retries, timeouts, circuit breakers, bulkheads, fallback behavior, and degraded modes all solve real problems.

But a pattern copied without system boundaries can create a new failure.

Example:

Checkout has provider timeouts.
The team adds retries everywhere.
The local queue drains for a moment.
The payment provider receives more traffic while unhealthy.
Other services using the provider slow down.
Checkout looks active but the wider system gets worse.

The missing mechanism is the boundary:

What dependency receives more load?
What user promise is protected?
What control surface limits harm?
What signal tells us the action is amplifying failure?

Repair:

Treat patterns as control surfaces.
Name the protected promise.
Name the blast radius.
Add caps, timeouts, and feedback delay.
Watch local and global signals after the action.

This is why "use a circuit breaker" is not enough.

A circuit breaker has settings, ownership, user behavior, support consequences, and recovery behavior.

The pattern is not magic.

It is a mechanism that needs a boundary.

Anti-Pattern 6: Reliability Heroics

Heroics happen when the system relies on exceptional human effort instead of a repeatable operating model.

It sounds admirable:

The on-call engineer knew the system deeply.
They remembered which script to run.
They manually repaired the stuck orders.
They stayed awake until the queue drained.

That person helped.

The anti-pattern is building a system where that person must exist every time.

Heroics hide toil.

They also hide missing design:

no support-safe order lookup
no bounded reconciler
no duplicate-risk signal
no clear incident handoff
no safe rollback test
no documented first action

Repair:

Turn repeated manual work into an artifact:
  signal
  runbook step
  automation
  support tool
  readiness check
  product decision

Do not automate everything immediately.

First, understand the work.

Then decide whether to automate, redesign, document, or accept the risk.

Synthesis Example: Diagnose A Mature-Looking Practice

Here is a reliability review packet for checkout.

At first glance, it looks mature:

SLO:
  checkout availability is 99.99%

Dashboard:
  CPU, memory, pod restarts, HTTP 500s, provider timeout rate

Alerts:
  page on CPU above 80%
  page on provider timeout rate above 2%
  page on any pod restart

Readiness:
  checklist has 42 items, all marked yes

Resilience:
  retries added to all provider calls

Economics:
  product asked for "best possible reliability"

Now diagnose it.

The corrected review packet is shorter and stronger:

Promise:
  users receive paid, declined, or safely pending within 2 minutes;
  one checkout attempt does not create duplicate charges

SLI:
  clear-result ratio
  duplicate-prevention risk signal
  pending-payment queue age

SLO:
  99.9% clear result within 2 minutes over 30 days
  duplicate-charge prevention treated as a stricter correctness guard

Alerting:
  page on fast burn of clear-result SLO
  page on duplicate-risk signal
  route provider timeout to dashboard unless user impact appears

Readiness:
  rollback can read pending-payment state
  support can inspect pending state safely
  degraded mode tested with provider timeout

Control:
  circuit breaker and retry caps have owners, limits, and feedback delay

Economics:
  99.99 is not purchased yet
  revisit if enterprise users require stricter checkout ambiguity limits

So far, the review skill is simple:

Do not ask whether the artifact exists.
Ask whether the artifact protects the promise through a working mechanism.

Retrieval Check

Match the anti-pattern to the missing mechanism.

Check: A team has a 99.99 SLO, but nobody can say what changes when the budget burns. What anti-pattern is most likely?

Think first, then reveal.

Answer: SLO decoration. The target exists, but it is not acting as a decision boundary for paging, release safety, prioritization, or product risk.

Trade-offs and Limits

Anti-pattern diagnosis helps teams avoid fake maturity.

It improves reviews because it asks for mechanisms, not labels.

It costs attention and humility.

Some practices that look impressive will need to be simplified, removed, or renamed.

The trade-off is this:

If you accept every reliability artifact at face value, you keep ceremony.
If you challenge every artifact too aggressively, you may slow useful work.
Good diagnosis asks for the smallest evidence that the practice changes a real decision.

Anti-pattern diagnosis does not replace deeper specialist work.

It will not design a full telemetry platform, incident program, chaos experiment, capacity model, or progressive delivery system.

It does help the foundation track do its job:

protect the user promise
choose meaningful signals
connect signals to decisions
act with known trade-offs
learn from feedback
spend reliability effort where it matters

You can see the boundary when the same anti-pattern appears across many teams.

At that point, the problem may not be one team's misunderstanding. It may be a platform gap, incentive problem, unclear ownership model, or missing product policy.

Common Confusions

Confusion: "If a practice is common, it must be mature"

Why it is tempting:

Common practices feel safe to copy.

Better model:

Common practices are useful only when the mechanism fits the service. A copied pattern still needs a promise, signal, decision, action, and feedback loop.

Confusion: "Anti-pattern diagnosis is blame"

Why it is tempting:

Calling something an anti-pattern can sound like accusing the team.

Better model:

The point is to improve the operating model. Most anti-patterns are reasonable attempts that lost their connection to user impact or decision-making.

Confusion: "Tools create reliability"

Why it is tempting:

Tools are visible, buyable, and easy to list in reviews.

Better model:

Tools support reliability when they help people measure, decide, act, and learn. A tool without that path is only inventory.

Confusion: "The fix is always more process"

Why it is tempting:

When reliability feels weak, adding process feels safer.

Better model:

Sometimes the fix is less process and a sharper mechanism: one better SLI, one clearer alert, one tested degraded mode, or one explicit accepted risk.

Transfer Challenge

Diagnose this order-history reliability packet.

Promise:
  not written down

Dashboard:
  API CPU, memory, pod restarts, request count

SLO:
  order history is 99.99% reliable

Alerts:
  page on any pod restart
  page on database CPU above 70%
  page on error rate above 1%

Readiness:
  checklist says dashboard, alert, owner, rollback, and runbook all exist

Runbook:
  "check logs and restart service if needed"

Pattern:
  client retries order-history reads up to 8 times

Product context:
  delayed receipts create many support contacts,
  but payment correctness is not affected

Write a diagnosis with:

1. Two anti-patterns you see.
2. The missing mechanism for each one.
3. One repaired reliability practice.
4. One trade-off or accepted risk.

Model answer:

Anti-pattern 1:
  Metrics theater.

Missing mechanism:
  The dashboard does not show whether users can see recent orders or receipts.

Repair:
  Define an SLI such as "recent orders visible within 5 minutes"
  and keep CPU as supporting evidence.

Anti-pattern 2:
  SLO decoration.

Missing mechanism:
  99.99 is not tied to user harm, support load, cost, or release decisions.

Repair:
  Choose a target based on receipt-delay harm, maybe 99.9% recent-order
  visibility within 5 minutes, then revisit if support contacts remain high.

Trade-off:
  The team may accept rare delayed receipts if payment correctness is safe,
  users get clear messaging, and support can see the order state.

A strong answer does not need to name every anti-pattern. It should show the habit: artifact -> missing mechanism -> repaired decision.

What Comes Next

The next lesson is the full synthesis review.

This lesson gives you the filter.

The synthesis lesson asks you to use the filter across the whole track:

promise
SLI
SLO
budget
alert
degraded mode
handoff
release decision
readiness evidence
control loop
economics
anti-pattern diagnosis

That synthesis will prepare for the final capstone, where you define and operate a reliability promise for one service.

Resources

• [BOOK] Site Reliability Engineering: Monitoring Distributed Systems
  • Link: https://sre.google/sre-book/monitoring-distributed-systems/
  • Focus: Use it to separate useful signals from monitoring inventory.
• [BOOK] Site Reliability Engineering: Service Level Objectives
  • Link: https://sre.google/sre-book/service-level-objectives/
  • Focus: Review how SLOs become decision tools instead of decorative targets.
• [BOOK] Site Reliability Workbook: Alerting on SLOs
  • Link: https://sre.google/workbook/alerting-on-slos/
  • Focus: Study alerting as a user-impact and action problem.
• [BOOK] Site Reliability Workbook: Production Readiness Reviews
  • Link: https://sre.google/workbook/production-readiness/
  • Focus: Compare checklist possession with readiness evidence.

Key Takeaways

• Reliability anti-patterns often start as reasonable practices that lose connection to user promise, evidence, decision, action, or learning.
• Metrics theater measures many things but misses the user-centered SLI.
• SLO decoration creates a target without an error-budget policy or product decision.
• Noisy paging wakes humans without urgent user impact or a safe first action.
• Checklist theater checks artifact possession instead of readiness evidence.
• A good reliability review asks what mechanism makes the practice protect the promise.