Alerting on Symptoms Instead of Causes

By the end of this lesson, you will be able to...

• Classify an alert as a user-visible symptom, an internal cause, or diagnostic context.

• Decide whether a signal should page someone, create a ticket, or stay on a dashboard.

• Rewrite a noisy alert into an actionable page tied to an SLI, an SLO, and a response.

Idea in one sentence: A page should wake a person for urgent user impact that needs action now, not for every internal thing that looks unhealthy.

Core Insight

In the previous lesson, the checkout team found a hidden cost: reliability was being paid for with repeated human work.

Now the team has a different problem.

At 02:13, the on-call phone starts buzzing.

Page 1:
  checkout-api CPU > 85% for 5 minutes

Page 2:
  payment-provider timeout rate > 8%

Page 3:
  pending checkout queue age > 10 minutes

Page 4:
  clear-checkout-result SLI burn rate is 12x for 10 minutes

Page 5:
  duplicate-authorization prevention hits increased

The on-call engineer is awake, but not yet helped.

Some of these signals may point to the same incident. Some may be normal side effects. Some may be useful only after the investigation has started. One may represent direct user harm.

The naive response is:

More alerts means more safety.

That idea feels responsible. Nobody wants to miss a failure.

But noisy paging creates a reliability problem of its own. It interrupts sleep, breaks focus, trains people to distrust alerts, and makes real incidents harder to see. It turns monitoring into another form of toil.

The better rule is:

Page on symptoms.
Use causes for diagnosis.

Plain meaning:

A symptom is evidence that the user promise is failing or will fail soon.

In this scenario:

Users not getting a clear checkout result within the promised time is a symptom.

Technical name:

An alert based on that user-visible failure is a symptom alert.

Plain meaning:

A cause is an internal condition that might explain a symptom.

In this scenario:

High CPU, provider timeouts, queue growth, database lock waits, or a bad deploy can all be causes.

Technical name:

An alert based only on an internal condition is a cause alert.

Cause signals are valuable. They help responders find the problem. They often belong on dashboards, in logs, in traces, or in lower-severity notifications.

But a cause does not automatically deserve a page.

The page is the most expensive alert channel because it asks a human to stop everything and act now. That channel should be protected.

The Naive Alerting Model

Many teams start with alerts like this:

CPU > 80%
memory > 85%
disk > 90%
error logs > 50/min
queue length > 1000
dependency timeout rate > 5%

These alerts are easy to create because the metrics are already available.

They also feel objective. A number crossed a threshold. The alert fired.

The problem is that most of these metrics are not the promise. They are clues about machinery.

High CPU can be harmless if the service is designed to run hot and latency is good.

Low CPU can be terrible if all workers are blocked waiting for a database lock.

A payment-provider timeout spike may be harmless if checkout has a safe pending state and users get a clear message.

A small number of duplicate-authorization prevention hits may be normal retry behavior. A sudden rise during a release may be a correctness risk.

The naive model treats every scary internal metric as equally urgent:

internal abnormality -> page

A better model separates three questions:

1. Is a user promise being broken, or close to being broken?
2. Does a human need to act now?
3. Is there a useful action or runbook for that human?

If the answer to all three is yes, a page may be right.

If not, the signal still matters, but it may need a different channel.

Paging, Tickets, and Diagnostic Context

Not every useful signal should interrupt a person.

Use channels as part of the design.

This is not about ignoring causes.

It is about placing causes where they help.

A page should say:

The promise is at risk.
Here is the symptom.
Here is the urgency.
Here is the first action.

A dashboard should say:

Here are the likely causes and supporting details.
Use them once you are investigating.

A ticket should say:

This condition is becoming risky.
Fix it before it becomes an incident.

The same metric can move between channels depending on threshold, time window, and action.

For example, disk usage at 70% may be dashboard context. Disk usage growing so fast that the queue store will fill in 15 minutes may be page-worthy because user impact is close and the action is urgent.

A Worked Alert Review

Review the five alerts from 02:13.

The service promise from earlier lessons is:

Users should receive a clear checkout result within 2 minutes:
paid, declined, or safely pending.

Users should not be charged twice for one checkout attempt.

The SLI is:

clear_checkout_result_ratio =
  checkout attempts with a clear final or pending-safe result within 2 minutes
  / all checkout attempts

The SLO is:

99.5% over 30 days

Now classify each alert.

Notice the difference between the first and fourth rows.

CPU > 85% says:

One internal resource is busy.

clear-checkout-result SLI burn rate is 12x says:

The service is currently failing the promise fast enough to spend too much
of the error budget.

The second message is a better page because it tells the responder why they are awake.

It also leaves room for multiple causes. The burn could come from CPU saturation, provider timeouts, a stuck queue, a bad deploy, or a database issue. The page does not pretend to know the cause before the investigation begins.

Check: If CPU is high and the SLI is healthy, should CPU page the on-call engineer?

Think first, then reveal.

Answer: Usually no. High CPU can be diagnostic context or a ticket if it is reducing safety margin. It becomes a page only when it predicts urgent user impact and there is a clear action to take now.

Designing the Page

A good page is not just a metric crossing a line.

It is a small operational message.

Bad page:

ALERT: checkout-api CPU high
value=87.3

Better page:

Page: Checkout clear-result SLO fast burn

Symptom:
  clear_checkout_result_ratio is burning 12x the allowed rate for 10 minutes

User impact:
  users may wait more than 2 minutes for paid, declined, or safely pending state

First checks:
  dashboard: checkout-result SLI by region
  dashboard: pending queue age
  dashboard: payment-provider timeout rate
  recent changes: checkout-api and payment-adapter deploys

First actions:
  if one region is affected, shift traffic away if safe
  if provider timeout dominates, enable pending-safe degraded mode
  if recent deploy correlates, roll back checkout-api
  if duplicate prevention hits spike, freeze automatic retries and escalate

The better page connects four things:

symptom -> user impact -> evidence -> action

That structure matters at 02:13.

The responder should not have to reverse-engineer why the alert exists. The alert should point to the promise, the urgency, and the first safe moves.

This also helps review.

After the incident, the team can ask:

Did this page represent real user impact?
Was the threshold early enough?
Was the first action useful?
Did the page include enough diagnostic context?
Did it fire too often for conditions that did not need a human?

That makes alerting part of the reliability control loop, not a pile of alarms.

SLO Burn Alerts

One common way to page on symptoms is to alert on SLO burn rate.

The idea is simple:

An error budget is the amount of failure you can tolerate.
A burn rate is how fast you are spending that budget.

If a service is allowed 0.5% bad checkout results over 30 days, a short spike may be acceptable. But a spike that continues at a high burn rate can consume the monthly budget quickly.

A burn alert asks:

If this continues, will we spend too much budget too fast?

That is better than paging on every individual error.

It also avoids waiting until the whole 30-day SLO is already lost.

For checkout:

Fast-burn page:
  clear checkout result is burning 12x budget for 10 minutes

Slow-burn ticket or page, depending on policy:
  clear checkout result is burning 2x budget for several hours

Fast burn usually means urgent investigation.

Slow burn may mean a sustained regression that needs planned work, a release pause, or a product decision. It might page during business hours or create a ticket, depending on the service and the team.

The exact thresholds are design choices. The principle is the important part:

The alert is tied to the promise and the budget, not merely to one machine.

Check: Why is "500 errors > 100 per minute" usually weaker than an SLO burn alert?

Think first, then reveal.

Answer: Raw error count lacks context. One hundred errors may be tiny during high traffic or severe during low traffic. An SLO burn alert relates failures to the allowed budget and the user-facing promise.

When Cause Alerts Are Appropriate

"Alert on symptoms" does not mean "never page on causes."

Sometimes a cause signal is direct enough, urgent enough, and actionable enough to page before users notice.

Examples:

queue storage will fill in 10 minutes
certificate expires tonight and renewal failed
primary database has no healthy replica
payment idempotency table is not accepting writes

These are still cause signals, but they have three important properties:

They are close to user impact.
They have little time margin.
They have a known action.

That is different from:

CPU is above 80%

The better cause page says:

The system is about to lose the ability to keep the promise.
Here is why.
Here is the time left.
Here is the action.

For example:

Page: Checkout idempotency writes failing

Impact risk:
  duplicate-charge protection may fail for new retry attempts

Evidence:
  idempotency write success rate below 99% for 5 minutes

Action:
  pause automatic payment retries
  route new checkouts to pending-safe state
  escalate to database on-call

That page is cause-based, but it is still promise-centered.

The user promise gives the cause its urgency.

Trade-offs and Limits

Symptom-based alerting improves paging quality.

It reduces alert fatigue, protects attention, and makes pages easier to justify. It also keeps the team focused on user promises instead of internal perfection.

It costs work.

You need honest SLIs, useful thresholds, runbooks, dashboards, and enough instrumentation to diagnose causes after a symptom page fires.

It can also miss early warning signs if the team removes every cause alert without replacing them with tickets, dashboards, or predictive pages. A symptom page should not be the only signal in the system.

The trade-off is:

Symptom alerts are better for paging.
Cause signals are still necessary for diagnosis and prevention.
The design work is choosing the right channel for each signal.

You can see the boundary when responders say:

"This page is real, but I do not know what to do."

That means the symptom is useful, but the response path is incomplete.

Or:

"This dashboard explains the incident, but it woke me up five times last week
when users were fine."

That means a diagnostic signal was put in the wrong channel.

Common Confusions

Confusion: "If it can cause an outage, it should page"

Why it is tempting:

Many causes can become user impact eventually.

Better model:

Page when the cause is urgent, close to user impact, and actionable. Otherwise use tickets, dashboards, capacity reviews, or release checks.

Confusion: "A symptom alert does not need cause details"

Why it is tempting:

The alert is correctly tied to user impact.

Better model:

The page should be symptom-first, not symptom-only. It should link to cause dashboards, recent changes, and first actions.

Confusion: "Fewer alerts always means better alerting"

Why it is tempting:

Noisy paging is painful.

Better model:

The goal is not fewer alerts by itself. The goal is fewer bad pages and clearer action for the pages that remain.

Confusion: "All SLO violations should page"

Why it is tempting:

SLOs feel important, so every violation feels urgent.

Better model:

Use burn rate, duration, severity, and service policy. A tiny budget spend may need review later. A fast burn needs immediate response.

Practice

Review these candidate alerts for the checkout service.

For each one, choose page, ticket, or dashboard, then explain why.

Alert A:
  checkout-api p95 latency is above 900 ms for 15 minutes
  clear checkout result SLI remains healthy

Alert B:
  clear checkout result SLI is burning 14x the allowed budget for 10 minutes
  recent deploy changed payment callback handling

Alert C:
  payment-provider timeout rate is 6% for 20 minutes
  pending-safe degraded mode is working and users receive clear pending state

Alert D:
  idempotency write failures are above 2% for 5 minutes
  automatic retries are still enabled

Model answer:

Alert A:
  Usually ticket or dashboard.
  Latency is worth watching, but the user promise is still healthy.
  Create a ticket if the trend reduces safety margin.

Alert B:
  Page.
  The user-facing promise is burning quickly, and a recent change gives a
  useful first investigation path.

Alert C:
  Usually dashboard or ticket.
  The provider is unhealthy, but the user-facing promise is protected for now.
  It may become a page if pending-safe mode starts missing its promise or if
  there is a runbook action that must happen immediately.

Alert D:
  Page.
  This is a cause signal, but it is close to a correctness promise.
  Duplicate-charge protection may fail, and the first action is clear:
  pause automatic retries or force pending-safe behavior.

Resources

• [BOOK] Site Reliability Engineering: Monitoring Distributed Systems
  • Link: https://sre.google/sre-book/monitoring-distributed-systems/
  • Focus: Pay attention to why alerting should be tied to symptoms, urgency, and human action.
• [BOOK] Site Reliability Workbook: Alerting on SLOs
  • Link: https://sre.google/workbook/alerting-on-slos/
  • Focus: Use the burn-rate examples to connect pages to error-budget consumption.
• [BOOK] Site Reliability Engineering: Service Level Objectives
  • Link: https://sre.google/sre-book/service-level-objectives/
  • Focus: Review how SLOs define the promise that alerting protects.

Key Takeaways

• A page should be tied to urgent user impact or imminent promise failure, not merely to an internal metric crossing a threshold.
• Cause signals are still valuable, but many belong in dashboards, tickets, logs, or traces instead of the paging channel.
• Good alert design connects symptom, user impact, evidence, and first action.
• SLO burn alerts are useful because they express how quickly the service is spending its allowed failure budget.
• Cause-based pages are appropriate when the cause is close to user impact, time-sensitive, and actionable.