Reliability Economics and Product Decisions
LESSON
Reliability Economics and Product Decisions
By the end of this lesson, you will be able to...
Explain why a reliability target is a product and cost decision, not only an engineering preference.
Compare under-reliability, useful reliability, and over-reliability for one service promise.
Write a short cost/risk trade-off note for choosing an SLO target.
Idea in one sentence: Reliability work should buy user trust where it matters most, while making the cost and accepted risk visible.
Core Insight
The checkout team now has a real operating model.
It has a promise:
Users should receive paid, declined, or safely pending within 2 minutes.
Users should not be charged twice for one checkout attempt.
It has signals, alerts, degraded modes, release decisions, readiness review, and control loops.
Then product asks a reasonable question:
Should checkout be more reliable?
The tempting engineering answer is:
Yes. More reliability is always better.
That answer is too simple.
Reliability is valuable, but it is not free. Buying more reliability can cost infrastructure, engineering time, product speed, operational attention, support design, and sometimes user experience in other areas.
The better question is:
How much reliability is worth buying for this promise,
for these users,
with these costs,
and with this accepted risk?
This is where reliability becomes a product decision.
It is still technical. The team still needs SLIs, SLOs, budgets, alerts, and control loops.
But the target itself is not purely technical.
It expresses a judgment about users, harm, money, reputation, opportunity cost, and product priorities.
Plain meaning:
Reliability economics means reasoning about what reliability costs, what reliability protects, and what risk the product chooses to accept.
In this scenario:
Moving checkout from a 99.5% clear-result SLO to a 99.99% clear-result SLO may protect more users, but it may require a second payment provider, more engineering time, stricter release gates, more support tooling, and slower feature delivery.
Technical name:
That judgment is a cost/risk trade-off.
The Naive Target: "As Close To 100% As Possible"
A naive reliability target sounds clean:
Checkout should be 100% reliable.
It sounds user-centered.
Nobody wants payment failures.
But "100%" hides the real decision.
Ask what the team would need to buy:
multiple payment providers
provider failover testing
stronger idempotency guarantees
more rollback compatibility work
larger on-call investment
more support workflows
stricter release gates
more capacity margin
more reconciliation tooling
more incident practice
Some of that may be worth it.
Some may not be.
The naive target also hides opportunity cost.
If the same engineering team spends six weeks reducing rare checkout ambiguity from 0.05% to 0.01%, it is not spending those six weeks on fraud reduction, accessibility, support tooling, international payment coverage, or the next product experiment.
Opportunity cost is not an excuse to ignore reliability.
It is the reason to choose reliability targets deliberately.
A Tiny Budget Calculation
Make the target visible.
Imagine the clear-result SLO is measured over 30 days.
A 30-day window has:
30 days * 24 hours * 60 minutes = 43,200 minutes
Now compare three targets:
| SLO target | Allowed bad minutes in 30 days | What it roughly means |
|---|---|---|
| 99.5% | 216 minutes | Several hours of allowed bad clear-result behavior. |
| 99.9% | 43.2 minutes | Less than one hour of allowed bad behavior. |
| 99.99% | 4.32 minutes | Only a few minutes of allowed bad behavior. |
This table is not the whole decision.
Real SLOs often count events, not minutes. One bad minute during peak traffic can hurt more users than one bad minute at night.
Still, the table shows the shape of the decision.
The difference between 99.5 and 99.9 is not just a nicer number.
It removes most of the error budget.
The difference between 99.9 and 99.99 is even sharper.
It leaves so little budget that normal deploy risk, dependency blips, queue drain time, and human response time may consume it quickly.
Check: Why is "99.99%" not automatically better than "99.9%"?
Think first, then reveal.
Answer: It may be better for some promises, but it buys a much smaller error budget at a higher cost. If users do not receive enough extra value from that improvement, the team may be over-spending engineering and product attention.
The Product Question Behind The Number
Reliability targets should start from user harm.
Ask:
What happens to the user when this promise fails?
Can the user retry safely?
Does the failure lose money, trust, data, or time?
Does the user know what happened?
Can support repair the situation?
How many users are affected during normal traffic?
How visible is the failure to the business?
For checkout:
Duplicate charge:
high harm, low tolerance
Truthful pending state:
medium harm if rare and clearly explained
higher harm if support cannot resolve it
Slow receipt display:
annoying, but lower harm if payment state is safe
Promo banner failure:
lower reliability need than payment correctness
These are not the same reliability problem.
A product can choose a strict target for duplicate-charge prevention and a looser target for a non-critical recommendation widget.
That is not hypocrisy.
It is good product judgment.
Different promises deserve different levels of protection.
Plain meaning:
Under-reliability means the service fails more often, or in more harmful ways, than users and the product can tolerate.
In this scenario:
If checkout often leaves users unsure whether they paid, the product is under-reliable even if the backend eventually reconciles correctly.
Technical name:
That is under-reliability: the target is too loose for the user harm.
Plain meaning:
Over-reliability means spending more reliability effort than the product value justifies.
In this scenario:
If the team builds multi-region failover and strict release freezes for a decorative checkout animation, it may be over-reliable.
Technical name:
That is over-reliability: the target is stricter than the promise deserves.
A Worked Cost/Risk Trade-off Note
The checkout team is choosing a clear-result SLO.
Three options are on the table:
Option A:
99.5% clear result within 2 minutes
Option B:
99.9% clear result within 2 minutes
Option C:
99.99% clear result within 2 minutes
Here is a concrete trade-off note.
| Question | Option A: 99.5% | Option B: 99.9% | Option C: 99.99% |
|---|---|---|---|
| User protection | Allows more ambiguous checkout results. | Strong protection for normal checkout. | Very strong protection, even for rare blips. |
| Error budget | Large enough for incidents and change. | Small but usable if controls are good. | Tiny; many normal risks become budget emergencies. |
| Engineering cost | Existing degraded mode and alerts may be enough. | Needs better support tooling, retry caps, provider monitoring, and release discipline. | Likely needs provider redundancy, deeper automation, stricter testing, and more staffing. |
| Product speed | High. | Moderate. | Lower, because changes need stronger proof and rollback safety. |
| Accepted risk | Users may see pending state too often during provider trouble. | Some pending state remains, but should be rare and handled. | Very little ambiguity accepted; expensive to maintain. |
| Good fit when | Checkout is lower volume or not business-critical yet. | Checkout is important and user trust matters. | Checkout is existential, regulated, or failure has very high harm. |
The team chooses Option B:
Target:
99.9% clear result within 2 minutes over 30 days
Why:
checkout is business-critical
users need a truthful result
duplicate-charge prevention is already strict
pending state is acceptable when rare and well explained
What we will buy:
support-safe pending-payment lookup
better queue-age alerting
stricter release pause during budget burn
provider timeout and retry caps
monthly review of pending-payment causes
What we will not buy yet:
active-active payment-provider failover
24/7 dedicated checkout incident commander
release freeze for every low-risk checkout copy change
Accepted risk:
during rare provider incidents, some users may receive a pending state
for longer than ideal, but duplicate charge prevention remains protected
Review trigger:
if pending-payment support contacts exceed agreed threshold
or enterprise customers name checkout ambiguity as a churn risk,
revisit the target and investment
Notice what this note does.
It does not say:
99.9 is the correct number because it feels mature.
It says:
This is the user harm.
This is the cost.
This is the risk we accept.
This is the signal that would make us reconsider.
That is reliability economics in practice.
What This Is Not
Reliability economics is not "spend less on reliability."
Sometimes the economic answer is:
Spend more.
If a failure can lose money, corrupt data, violate regulation, endanger users, or destroy trust, the right target may be strict and expensive.
Reliability economics is also not "let product override engineering."
Engineering owns the evidence:
what the system can measure
what the system can control
what failure modes exist
what a target would cost
what risks are easy or hard to mitigate
Product owns part of the value judgment:
which users matter most here
what harm the product cannot accept
what trade-off is worth making
what business risk is acceptable
what product speed matters
Good reliability decisions need both.
If engineering chooses targets alone, it may optimize for beautiful systems that do not match product value.
If product chooses targets alone, it may ask for numbers the system cannot honestly support.
The Cost Curve
Reliability cost is often not linear.
Going from poor reliability to decent reliability can be cheap:
add a user-centered SLI
set a realistic SLO
fix the worst timeout
add a degraded mode
write a rollback-safe release rule
Going from decent reliability to excellent reliability costs more:
remove single points of failure
test dependency failure regularly
build support tooling
reduce toil
improve automation
make rollback compatible with new state
Going from excellent reliability to extreme reliability can be very expensive:
multi-region operation
multiple providers
strict change controls
deep capacity reserves
dedicated on-call staffing
formal incident exercises
complex data repair tools
Each step may be valuable.
But each step should earn its cost.
This is the central trade-off: stricter reliability targets reduce accepted user harm, but they also consume money, time, attention, and product flexibility.
The signal to watch is mismatch.
Under-reliability shows up as:
support volume
lost trust
churn
manual repair
incident frequency
budget burn that product cannot tolerate
users unable to complete important work
Over-reliability shows up as:
teams blocked by risk that users barely notice
expensive redundancy protecting low-value paths
SLOs that trigger emergency process for harmless failures
engineering time spent polishing rare cases while common user pain waits
Neither mismatch is good.
The goal is not maximum reliability everywhere.
The goal is the right reliability for each promise.
Check: A marketing banner fails for 20 minutes, but checkout, payment, and order history still work. Should the banner usually have the same SLO as payment correctness?
Think first, then reveal.
Answer: Usually no. The banner may matter, but payment correctness protects higher user harm. The banner can often have a looser target and a simpler degraded mode.
Common Confusions
Confusion: "The highest SLO is the most professional SLO"
Why it is tempting:
High numbers look mature and ambitious.
Better model:
A professional SLO matches user harm, system capability, and product value. A target that is too strict can create false emergencies and slow useful work.
Confusion: "Business context weakens engineering rigor"
Why it is tempting:
Engineers may worry that cost discussions are a way to excuse unreliable systems.
Better model:
Business context makes rigor sharper. It tells the team which promises deserve the strongest evidence, controls, and investment.
Confusion: "If users complain, the target is automatically too low"
Why it is tempting:
Complaints are painful and visible.
Better model:
Complaints are evidence, but the team still needs to classify the harm. The fix might be a stricter SLO, clearer degraded messaging, better support tooling, or a different product promise.
Confusion: "Accepted risk means ignored risk"
Why it is tempting:
The phrase can sound like giving up.
Better model:
Accepted risk should have an owner, reason, signal, and review trigger. Hidden risk is ignored risk. Named risk is a decision.
Practice
Write a cost/risk trade-off note for this service:
Service:
order history
Promise:
users can see recent orders and receipts
Current behavior:
order history is available 99.5% of the time
during dependency incidents, recent orders may be missing for up to 20 minutes
payment correctness is not affected
support receives many contacts when receipts are delayed
Proposal:
raise order history to 99.99%
Your note should include:
- User harm.
- Candidate target.
- What the stricter target would buy.
- What it would cost.
- Accepted risk.
- Review trigger.
Model answer:
User harm:
delayed receipts create anxiety and support contacts,
but payment correctness and order creation are safe
Candidate target:
99.9% recent-order visibility within 5 minutes
instead of jumping directly to 99.99%
What this buys:
fewer anxious users
clearer support answers
faster detection when order-history lag is too high
Cost:
receipt cache
lag SLI
support-safe order-state view
alert on recent-order lag
some release discipline around order-history changes
Accepted risk:
rare dependency incidents may still delay recent orders,
but users receive a clear message and support can see state
Review trigger:
revisit 99.99% if enterprise users require it,
if delayed receipts cause measurable churn,
or if support load remains high after the 99.9% investment
A strong answer does not simply choose the biggest number. It connects the number to user harm, cost, and a signal that would change the decision.
Resources
- [BOOK] Site Reliability Engineering: Embracing Risk
- Link: https://sre.google/sre-book/embracing-risk/
- Focus: Read for the idea that reliability targets should reflect explicit risk tolerance.
- [BOOK] Site Reliability Engineering: Service Level Objectives
- Link: https://sre.google/sre-book/service-level-objectives/
- Focus: Connect user expectations, SLOs, and the cost of stricter targets.
- [BOOK] Site Reliability Workbook: Implementing SLOs
- Link: https://sre.google/workbook/implementing-slos/
- Focus: Use it for practical SLO design and stakeholder discussion.
- [BOOK] Site Reliability Workbook: Error Budget Policy
- Link: https://sre.google/workbook/error-budget-policy/
- Focus: Study how reliability targets become operating and product decisions.
Key Takeaways
- A reliability target is a product and cost decision because it chooses how much user harm the product will prevent and what it will pay to prevent it.
- Higher SLO numbers reduce error budget quickly; 99.99% is not just "a little better" than 99.9%.
- Under-reliability accepts more user harm than the product can tolerate; over-reliability spends too much attention on promises that do not need that level of protection.
- Engineering should bring evidence about system cost and risk; product should help judge user harm and business value.
- Good reliability economics names the target, cost, accepted risk, owner, and review trigger.
← Back to Reliability Engineering Foundations