SLIs: Measuring What Users Feel

By the end of this lesson, you will be able to...

• Choose SLIs that approximate a user promise instead of merely describing internal machinery.

• Trace a checkout request and decide which events count as good, bad, valid, or excluded.

• Reject convenient metrics when they are too far away from user-visible reliability.

Idea in one sentence: An SLI is a carefully chosen signal that turns a user promise into countable evidence, close enough to the user's experience to guide reliability decisions.

Core Insight

In the previous lesson, we wrote a checkout promise:

For normal card payments in the primary region, valid checkout requests should
produce a clear success or failure result within 3 seconds, and one checkout
attempt should not create duplicate charges.

Now the on-call team needs to operate that promise.

At 10:12, support reports:

Users say checkout is "stuck."
Some users click Pay twice.
One customer asks whether they were charged.

The dashboards show many numbers:

pod CPU: 71%
memory usage: stable
load balancer 5xx rate: 0.2%
POST /checkout p95 latency: 2.8s
payment-provider timeout rate: 6%
checkout result page success rate: 92%
duplicate authorization attempts: rising

Which number tells us whether the promise is holding?

Not every metric is an SLI.

Plain meaning:

An SLI is a signal we use as evidence for a reliability promise.

In this scenario:

The promise is about clear checkout results, time, and duplicate charges. Good SLIs should be close to those user-visible outcomes.

Technical name:

That signal is a service level indicator, usually shortened to SLI.

The word "indicator" is doing real work. An SLI does not perfectly contain the whole user experience. It points at it. It is a useful approximation.

If the indicator is too far from the user, it can look healthy while users suffer. If the indicator is too vague, the team cannot make decisions. If the indicator is too expensive to collect, the team may stop trusting it or checking it.

Good reliability work starts by asking:

What user experience did we promise?
What event can we count?
Which counted events are good?
Which counted events are valid for this promise?
What does the signal miss?

The Naive Dashboard

The naive idea is:

Collect many metrics. More metrics means more reliability.

This feels sensible. If one dashboard is good, ten dashboards must be better.

But during an incident, too many unrelated signals can make the team slower. A dashboard can be full and still not answer the question.

For checkout, these metrics are useful context:

CPU
memory
pod restart count
database connections
queue depth
payment-provider timeout count
HTTP 5xx rate

They may help explain a problem. They may predict a problem. They may identify a component under pressure.

But most of them do not directly say:

Did the user get a clear checkout result?
Was it fast enough?
Was the user protected from duplicate charge behavior?

The naive dashboard breaks because internal health is not the same as user outcome.

Imagine two situations:

Situation A:
  CPU is high.
  Checkout still returns clear results within 3 seconds.
  No duplicate charges happen.

Situation B:
  CPU is normal.
  Payment callbacks are delayed.
  Users see a spinner and retry.
  Duplicate authorization attempts rise.

Situation B is worse for the checkout promise, even though the server looks calmer.

The lesson is not "ignore CPU."

The lesson is:

Use internal metrics as supporting evidence.
Use SLIs as promise evidence.

What Makes a Good SLI

A useful SLI has four pieces.

First, it has a user-facing event.

For checkout:

A valid user submits POST /checkout.

Second, it has a good condition.

For checkout:

The user receives a clear success or failure result within 3 seconds.

Third, it has a valid population.

For checkout:

Normal card payments in the primary region.
Exclude invalid requests, test traffic, and traffic outside the promise boundary.

Fourth, it has a measurement source.

For checkout:

Server-side request logs plus result-page events, joined by checkout attempt id.

That gives the SLI a countable shape:

good events / valid events

For example:

good:
  valid checkout attempts that receive a clear result within 3 seconds

valid:
  valid checkout attempts for normal card payments in the primary region

SLI:
  good / valid

This looks simple. The hard part is deciding what "good" and "valid" mean without lying to yourself.

If you count only HTTP 200 responses, you may miss users who saw a timeout after the payment provider accepted the charge.

If you count only server latency, you may miss client-side result-page failures.

If you include invalid requests from broken clients, the SLI may punish the service for traffic it never promised to handle.

If you exclude too much, the SLI becomes flattering. It says the service is good because it stopped looking at the painful cases.

Good SLI design is careful honesty.

A Worked Trace

Trace five checkout attempts during a small incident.

The promise is:

Valid card checkout should produce a clear result within 3 seconds and avoid duplicate charge behavior.

Events:

Now calculate the SLI for this small window:

valid events = A, B, C, E = 4
good events = A, B = 2

SLI = good / valid = 2 / 4 = 50%

The same window can fool a weaker metric:

HTTP 5xx rate:
  A: no 5xx
  B: no 5xx
  C: no 5xx if the request stayed open or timed out client-side
  D: maybe 4xx
  E: no 5xx

The 5xx dashboard might look calm.

The SLI says the promise is in trouble.

That is the point. A good SLI is not the easiest thing to count. It is the count that best approximates the experience the team promised to protect.

Check: Why does attempt B count as good even though the payment failed?

Think first, then reveal.

Answer: The reliability promise is not "all cards are accepted." It is that valid checkout attempts receive a clear result within 3 seconds and avoid unsafe ambiguity. A clear card-declined result can keep the reliability promise.

Signals Close to the User

When choosing SLIs, ask how close the signal is to the user's task.

For checkout:

Distance from the user is not automatically bad.

Low-level signals can be excellent early warning signs. CPU, queue depth, connection pool saturation, and dependency errors can tell you why the promise may break soon.

But do not confuse early warning with promise evidence.

Use this mental split:

SLI:
  "Is the promise holding?"

Diagnostic signal:
  "Why might the promise be failing?"

Control signal:
  "Should we change behavior now?"

The same metric can play different roles depending on the promise. For an internal batch system, queue age may be close to the user because users care when reports become available. For checkout, queue age may be diagnostic unless it directly affects clear checkout results.

Context decides.

What The System Knows

A checkout SLI is not free. The system must record enough evidence.

A useful event might look like this:

checkout_attempt_id: chk_4821
region: primary
payment_method: card
request_valid: true
result_seen_by_user: success
result_latency_ms: 920
duplicate_authorization_attempted: false
source: server_and_client_joined

This event lets the team count the promise.

But many systems start with only this:

path: /checkout
status: 200
duration_ms: 700
pod: checkout-api-7c9d

That is easier to collect. It is also less honest about the promise.

It cannot tell whether the user saw a clear result page. It cannot tell whether a duplicate authorization attempt happened. It cannot tell whether a fast API response led to an ambiguous UI state.

So SLI work often creates instrumentation work.

That is a trade-off:

Better SLIs improve judgment.
Better SLIs may require better event design.

The next track on observability will go deeper into telemetry pipelines. Here, the reliability lesson is simpler: if the service cannot observe the promised experience, the team cannot operate the promise well.

Investigation Path

Use SLIs during an incident by starting from the promise and moving inward.

For the checkout report:

Symptom:
  users say checkout is stuck

Promise question:
  are valid attempts getting a clear result within 3 seconds?

SLI check:
  clear-result SLI dropped from 99.4% to 91.8%

Breakdown:
  failures concentrated in primary region, card payments, payment-provider calls

Diagnostic signals:
  payment-provider timeout rate rose from 0.4% to 6%
  duplicate authorization attempts rose
  API 5xx rate stayed low

Operational decision:
  treat this as user-impacting even though 5xx is quiet

Notice the direction.

We did not start with every dashboard and hope one felt important. We started with the promise, checked the SLI, then used diagnostic signals to explain the SLI movement.

This direction reduces noise.

It also makes handoff easier. If the team later escalates to incident management, the handoff can say:

The checkout clear-result SLI is failing for card payments in the primary region.
Payment-provider timeouts appear correlated.
Users are retrying, and duplicate authorization attempts are rising.

That is much more useful than:

Some graphs look weird.

Check: If POST /checkout p95 latency is 250 ms but the clear-result SLI is falling, which signal should drive the reliability decision?

Think first, then reveal.

Answer: The clear-result SLI should drive the reliability decision because it is closer to the promise. The latency metric becomes diagnostic: it tells us the API handler may not be the slow part, so we should inspect result rendering, callbacks, client behavior, or downstream ambiguity.

Trade-offs and Limits

SLIs improve reliability work because they turn vague concern into evidence.

They help teams decide:

Is there user impact?
Which promise is failing?
Is the failure inside the boundary we promised?
Which users or workflows are affected?
Should we alert, degrade, pause change, or investigate quietly?

They also have limits.

First, an SLI is an approximation. It can miss parts of the experience. A checkout SLI based only on server events may miss a broken frontend result page. A client-side event may miss users who close the page before reporting success or failure.

Second, an SLI can be gamed accidentally. If the team excludes "slow payment provider" traffic from the valid population, the SLI may improve while users still suffer. Exclusions need strong reasons.

Third, an SLI can be too expensive. Joining server events, client events, and payment events may require careful identifiers and privacy review. Perfect measurement can become its own project.

Fourth, an SLI can create tunnel vision. If checkout has one SLI for fast success but none for duplicate charges, the team may optimize speed while missing safety.

The trade-off is:

A close SLI is harder to design and collect.
A distant metric is easier to collect but easier to misread.

Good teams usually keep both:

few user-centered SLIs for decisions
many diagnostic signals for explanation

Common Confusions

Confusion: "Every metric on the dashboard is an SLI"

Why it is tempting:

Dashboards often mix user, service, infrastructure, and dependency metrics in one place. They all look official.

Better model:

An SLI is selected because it approximates a service promise. Other metrics can still be useful as diagnostic signals.

Confusion: "Only successful business outcomes count as good"

Why it is tempting:

It feels natural to count only successful purchases, successful uploads, or successful searches.

Better model:

Reliability often includes clear failure. A declined card, a rejected invalid upload, or a no-results search can be a good reliability event if the service gives the user a correct and timely answer.

Confusion: "The best SLI is always the closest user signal"

Why it is tempting:

User-centered measurement is the goal, so the closest signal sounds automatically best.

Better model:

The closest signal still needs to be trustworthy, timely, and affordable. A client event that arrives late, drops often, or cannot be joined to backend state may need a simpler companion signal.

Confusion: "A green SLI means no one is unhappy"

Why it is tempting:

SLIs compress many events into one number.

Better model:

A green SLI means the measured promise is inside its acceptable range. It does not prove every user is happy, every segment is healthy, or every failure mode is covered.

Practice

Choose SLIs for an image upload service.

Promise:

For valid profile images under 5 MB, the user should receive a clear upload
accepted or rejected result within 4 seconds. If thumbnails are delayed, the
profile should show a safe placeholder instead of making the upload look lost.

Candidate signals:

A. percentage of valid uploads with a clear result within 4 seconds
B. average CPU on image-processing workers
C. percentage of accepted uploads that have a durable original image within 1 minute
D. number of log lines containing "thumbnail"
E. percentage of profile views that show either a thumbnail or the expected placeholder

Pick two strong SLIs and one diagnostic signal.

Model answer:

Strong SLI 1:
  A, because it measures the clear-result part of the promise.

Strong SLI 2:
  E, because it measures whether delayed thumbnails still produce an acceptable
  user-visible profile state.

Diagnostic signal:
  B, because worker CPU may explain thumbnail delay, but users do not directly
  experience worker CPU.

C may also be a strong SLI if the promise includes durable original storage. It is slightly less directly visible than A or E, but it protects an important hidden part of the user outcome.

D is weak by itself. Log volume can help debugging, but the number of log lines does not say whether users got a clear upload result.

Resources

• [BOOK] Site Reliability Engineering: Service Level Objectives
  • Focus: Read for the relationship between user journeys, SLIs, and the difference between indicators and objectives.
• [BOOK] The Site Reliability Workbook: Implementing SLOs
  • Focus: Use the practical SLI examples and the warning that convenient measurements can be misleading.
• [DOC] Google Cloud Architecture Center: Defining SLIs
  • Focus: Compare request-based, window-based, and freshness-style indicators.

Key Takeaways

• An SLI is promise evidence: a signal chosen because it approximates what users feel.
• A useful SLI defines good events, valid events, boundaries, and a measurement source.
• Internal metrics like CPU, queue depth, and dependency timeouts are valuable diagnostics, but they are not automatically SLIs.
• Clear failures can count as good reliability events when the promise is to give a correct and timely result.
• Good reliability decisions start from the promise, check SLIs, then use diagnostic signals to explain what is happening.