SLOs and Error Budgets

By the end of this lesson, you will be able to...

• Turn an SLI into an SLO by choosing a target, a measurement window, and a boundary.

• Calculate a small error budget and explain what it means for release and reliability decisions.

• Compare SLO targets that are too strict, too loose, or useful enough to operate.

Idea in one sentence: An SLO turns a reliability signal into a decision rule by saying how much imperfection is acceptable over a time window, and the error budget makes that imperfection visible.

Core Insight

The previous lesson gave us a checkout SLI:

good events:
  valid checkout attempts that produce a clear result within 3 seconds

valid events:
  valid card checkout attempts in the primary region

SLI:
  good / valid

Now the team has a harder question:

How good is good enough?

At first, the tempting answer is:

100%.

That answer feels responsible. Nobody wants checkout to fail.

But 100% is not a decision rule. It is a wish.

A real checkout system depends on networks, databases, payment providers, browsers, deploys, queues, caches, and humans. Some requests will fail. Some will be slow. Some will be ambiguous. Some failures are worth preventing at high cost. Some are not worth stopping every product change for.

The design problem is not:

Can we make failure impossible?

The design problem is:

What level of reliability should we promise, over what window, for which users,
and what should change when we are spending that promise too quickly?

Plain meaning:

An SLO is the reliability target the team agrees to operate against.

In this scenario:

The checkout team might decide that at least 99.5% of valid card checkout attempts in the primary region should produce a clear result within 3 seconds over a rolling 28-day window.

Technical name:

That target is a service level objective, usually shortened to SLO.

The gap between the target and perfection is the error budget.

For a 99.5% SLO:

allowed imperfection = 100% - 99.5% = 0.5%

That 0.5% is not a permission to be careless.

It is the amount of unreliability the service can spend before the team should change behavior.

The Promise We Need to Keep

Use the same checkout promise:

For normal card payments in the primary region, valid checkout requests should
produce a clear success or failure result within 3 seconds, and one checkout
attempt should not create duplicate charges.

The SLI gives evidence:

clear-result SLI = good checkout attempts / valid checkout attempts

The SLO adds a target:

clear-result SLO = 99.5% over 28 days

A full SLO statement should be boringly explicit:

For valid card checkout attempts in the primary region, at least 99.5% should
produce a clear success or failure result within 3 seconds, measured over a
rolling 28-day window from joined server and result-page events.

This statement has the pieces a team needs:

SLI:
  clear-result rate

target:
  99.5%

window:
  rolling 28 days

population:
  valid card checkout attempts in the primary region

measurement source:
  joined server and result-page events

Without these pieces, people can agree in a meeting and still mean different things.

"Checkout should be reliable" could mean 99%, 99.9%, only API success, only web users, only business hours, or only the current dashboard.

An SLO removes some of that fog.

The Naive Design

The naive design is:

Set the target as high as possible.
Page whenever the signal dips.
Declare victory when the graph is green.

This is tempting because high targets sound serious.

But an SLO that is too strict can harm the product.

Suppose checkout has 100,000 valid attempts in 28 days.

Compare three targets:

The right target is not the biggest number the team can write.

The right target is the number that matches user expectations, product risk, engineering cost, and the service's current maturity.

If the target is too strict, the team may stop useful releases for tiny changes in a noisy signal. People learn to ignore the SLO because it always complains.

If the target is too loose, users suffer while the dashboard still says the service is inside the objective. People learn to distrust the SLO because it misses obvious pain.

A useful SLO sits in the uncomfortable middle:

strict enough to protect users
realistic enough to guide behavior
clear enough to settle arguments

A Worked Budget

Now calculate the error budget.

Assume:

valid checkout attempts in 28 days: 100,000
SLO target: 99.5%
allowed bad percentage: 0.5%

The budget is:

100,000 * 0.005 = 500 bad checkout attempts

That means the service can have up to 500 valid attempts that do not receive a clear result within 3 seconds during the 28-day window before it misses the SLO.

Now trace a week:

For the first three days, the service is spending slowly.

On day 4, something changes. Maybe a payment-provider timeout increased. Maybe a release changed result-page behavior. Maybe retry logic created ambiguity.

By day 7, the service has spent more than the 28-day budget.

The budget turns an emotional statement:

"Checkout feels bad this week."

into an operating statement:

"We spent the 28-day error budget in 7 days. The clear-result promise is being consumed too quickly."

That operating statement can drive decisions.

pause risky releases
focus engineering time on the failure mode
degrade gracefully for the payment provider
improve result-page state handling
adjust alerting around fast budget burn

Check: If the team has 100,000 valid events and a 99.5% SLO, how many bad events can fit inside the budget?

Think first, then reveal.

Answer: 500 bad events. A 99.5% target allows 0.5% bad events, and 0.5% of 100,000 is 500.

Design Alternatives

SLO design is a product and engineering choice.

Consider three possible checkout SLOs.

Option A: 100% clear result within 3 seconds

This sounds brave.

It is usually brittle.

One bad event breaks the objective. The team cannot distinguish a tiny measurement glitch from meaningful user harm. Every release becomes suspect. The SLO is too strict to help with prioritization.

Use this only when the domain truly requires near-zero failure and the organization is willing to pay for it with architecture, process, testing, redundancy, and slower change.

Option B: 95% clear result within 3 seconds

This sounds relaxed.

It is probably too loose for checkout.

Out of 100,000 attempts, 5,000 could be unclear or too slow while the service still meets the objective. For a payment path, that may create support load, lost revenue, and user distrust.

The SLO would be green while many users are reasonably unhappy.

Option C: 99.5% clear result within 3 seconds over 28 days

This may be a useful starting point.

It allows 500 bad attempts in 100,000. That is not perfect, but it creates a real budget. It gives the team room to ship, learn, and absorb small failures. It also creates a clear signal when failures are too frequent.

This target may still be wrong.

Maybe checkout needs 99.9%. Maybe mobile networks make 3 seconds too strict for some regions. Maybe duplicate authorization deserves a separate SLO because it is rarer but more harmful.

The point is not that 99.5% is magic.

The point is that the SLO is now a design object the team can review.

Error Budget as a Decision Rule

The error budget is useful because it connects reliability to behavior.

Without a budget, teams often argue from mood:

Product:
  "We need to ship."

Engineering:
  "The service feels risky."

Support:
  "Users are complaining."

Leadership:
  "Can we do both?"

An error budget does not remove judgment. It gives judgment a shared reference.

For example:

Budget mostly intact:
  Continue normal releases.
  Watch for regressions.
  Invest in reliability work according to normal priority.

Budget burning quickly:
  Slow risky changes.
  Investigate the failure mode.
  Add mitigation or rollback criteria.
  Escalate if user harm is active.

Budget exhausted:
  Stop non-urgent risky launches for this service.
  Focus on restoring the promise.
  Review what consumed the budget.

This is where the phrase "error budget" can be misleading.

It is not a budget to spend carelessly.

It is a budget for risk.

If the service spends it slowly, the team has evidence that current reliability and change practices are roughly compatible. If the service spends it quickly, the team has evidence that something must change.

Check: A team misses the SLO but says, "Most failures came from a dependency, not our code." Should the budget ignore those failures automatically?

Think first, then reveal.

Answer: No. If users experienced the failed promise, the budget should usually count it. The dependency may explain the failure and guide mitigation, but the user promise crosses dependency boundaries unless the SLO explicitly and honestly excludes that case.

Trade-offs and Limits

SLOs improve reliability work because they create explicit decision boundaries.

They help teams answer:

How reliable are we trying to be?
How much imperfection did we agree is acceptable?
Are we spending risk faster than expected?
Should reliability work outrank feature work right now?

But SLOs have limits.

First, an SLO can be precise and still wrong. If it measures the wrong SLI, the target does not matter. A 99.99% objective on pod uptime will not protect users from ambiguous checkout results.

Second, an SLO can hide segments. The overall checkout SLO may be green while one region, browser, payment method, or customer tier is suffering. Important promises may need segmented views or separate SLOs.

Third, an SLO can create bad incentives. If teams are punished for missing SLOs, they may choose loose targets or exclude painful traffic. The budget should encourage honest trade-offs, not creative accounting.

Fourth, an SLO is not a complete incident policy. Fast budget burn can trigger attention, but humans still need to judge severity, safety, communication, and mitigation.

The trade-off is:

Strict SLOs protect users but cost more and can slow change.
Loose SLOs preserve speed but may normalize user pain.
Useful SLOs make that trade-off visible.

Common Confusions

Confusion: "The SLO is the same thing as the SLI"

Why it is tempting:

Both names are short, and both appear on the same dashboard.

Better model:

The SLI is the measurement. The SLO is the target for that measurement. "Clear-result rate" is an SLI. "Clear-result rate should be at least 99.5% over 28 days" is an SLO.

Confusion: "The error budget is extra failure we are allowed to create"

Why it is tempting:

The word budget sounds like permission to spend.

Better model:

The error budget is a risk signal. It tells the team how much room remains before the service misses its promise. Spending it may be acceptable; spending it fast should change behavior.

Confusion: "A missed SLO always means someone did something wrong"

Why it is tempting:

Targets feel like grades.

Better model:

A missed SLO means the service promise was not met as designed. The cause may be code, dependency behavior, capacity, product assumptions, measurement mistakes, or unrealistic targets. The useful question is what decision should change.

Confusion: "One SLO can represent the whole service"

Why it is tempting:

One number is easy to present.

Better model:

One number may be useful for a narrow promise. A service with latency, correctness, durability, and safety promises may need several SLOs, each tied to a real user outcome.

Practice

Design a first SLO for the image upload promise from the previous lessons.

Promise:

For valid profile images under 5 MB, the user should receive a clear upload
accepted or rejected result within 4 seconds. If thumbnails are delayed, the
profile should show a safe placeholder instead of making the upload look lost.

Candidate SLI:

valid uploads with a clear accepted or rejected result within 4 seconds
/
valid upload attempts for profile images under 5 MB

Traffic:

valid uploads in 28 days: 40,000

Pick:

1. An SLO target.
2. A measurement window.
3. The resulting error budget in bad upload attempts.
4. One decision the team should make if the budget is spent in the first week.

Model answer:

SLO:
  99.2% of valid profile-image uploads should produce a clear accepted or
  rejected result within 4 seconds over a rolling 28-day window.

Error budget:
  100% - 99.2% = 0.8%
  40,000 * 0.008 = 320 bad upload attempts

Decision if spent in week one:
  Pause risky upload pipeline changes and investigate the delay or ambiguity.
  Keep urgent safety fixes moving, but require explicit rollback or mitigation
  plans for changes that could affect upload clarity.

Your target can differ. The important test is whether you can explain the trade-off. A stricter target protects users more but costs more. A looser target preserves speed but accepts more unclear uploads.

Resources

• [BOOK] Site Reliability Engineering: Service Level Objectives
  • Focus: Use this chapter for the relationship between SLIs, SLOs, user journeys, and reliability targets.
• [BOOK] Site Reliability Engineering: Embracing Risk
  • Focus: Read for the idea that reliability targets are explicit risk decisions, not attempts at infinite perfection.
• [BOOK] The Site Reliability Workbook: Implementing SLOs
  • Focus: Use the practical examples of SLO documents, measurement windows, and error budget policy.

Key Takeaways

• An SLI measures promise evidence; an SLO sets the target for that measurement.
• The error budget is the allowed gap between the SLO target and perfection over a window.
• A useful SLO includes a target, window, population, SLI, and measurement source.
• Error budgets help teams decide when to keep shipping, slow down, investigate, or restore the promise.
• SLO design is a trade-off: too strict creates noise and cost; too loose hides user pain.