Decision Quality Is Not Outcome Quality

By the end of this lesson, you will be able to...

• Separate decision quality, execution quality, outcome quality, and luck in a real review.

• Judge a decision using the evidence that was available before the result was known.

• Recognize when “we were unlucky” hides weak investigation or ignored warning signs.

Idea in one sentence: A decision should be judged by whether its process was reasonable before the outcome was known, while the outcome should be used as new evidence rather than as the whole verdict.

Core Insight

A checkout team must release a caching change before a weekend sales campaign. The change passed its tests and reduced latency in a small canary. The team chooses a staged rollout with a rollback trigger.

During the rollout, a cloud zone fails. Traffic moves to the remaining zones, the cache overloads, and checkout becomes unavailable for twelve minutes.

On Monday, the incident review begins with a blunt sentence:

The release was a bad decision. Look at what happened.

The outage matters. It may reveal a missing load test, a weak rollback trigger, or a dependency the team should have understood. But the outage alone does not tell us whether the original decision was poor.

An outcome is observed after uncertainty has resolved. A decision is made before that resolution, with partial information. If we collapse those two moments, we teach ourselves the wrong lesson.

Why the Outcome Takes Over the Story

Once we know an outcome, it feels inevitable. We can see the failed zone, the overloaded cache, and the twelve-minute outage. The causal path is now bright and easy to describe.

Before the release, that path was only one branch among many. The campaign might have been quiet. The zone might have stayed healthy. The cache might have reduced latency exactly as the tests predicted. Other failures might have mattered more.

Judging the choice only by its result is often called outcome bias. In decision practice, people also use the word resulting: treating a good result as proof of a good decision, or a bad result as proof of a bad one.

The mistake works in both directions:

• A careful decision can produce a bad outcome because a low-probability risk occurs.
• A careless decision can produce a good outcome because the ignored risk does not occur this time.

If the team rewards only winners, lucky recklessness looks like skill. If it punishes only losers, careful experiments become politically dangerous.

Four Things Happened, Not One

To review the checkout release, separate four layers.

1. Information quality

What evidence did the team have? What remained uncertain? Which additional evidence was reasonably obtainable before the deadline?

Information quality is not the same as information quantity. Fifty dashboards do not help if none tests the important failure mode. The question is whether the evidence was relevant, credible, and proportional to the stakes.

2. Decision quality

Given that information, did the choice fit the goal, constraints, uncertainty, and known risks? Were assumptions visible? Did the team define what would make it stop or reconsider?

In plain English, decision quality asks:

Did this choice make sense at the time, for defensible reasons?

It does not ask whether the team predicted everything.

3. Execution quality

Was the chosen plan carried out as intended? A sensible staged rollout can still be executed badly. Someone may bypass the canary, misconfigure the rollback, or ignore the trigger.

That is not necessarily bad luck. It may be an execution failure.

4. Outcome quality

What actually happened, and how desirable was it? Revenue, latency, errors, customer harm, recovery time, and trust all belong here.

Outcome quality matters. The distinction does not make outcomes unimportant. It prevents the outcome from erasing the earlier layers.

The Decision Review Card

Let us reconstruct the release using only what could have been written before the campaign.

Now add the execution and outcome:

The review can now ask precise questions.

1. Was the zone-failure risk reasonably knowable?
2. Was a failure test affordable before the campaign?
3. Did the team underestimate correlation between zone loss and cache load?
4. Was the delayed alert an execution problem, a monitoring-design problem, or both?
5. Would the same decision still be defensible after updating the model with this evidence?

Notice what changed. “The outage proves the release was bad” became several testable claims.

The original staged-release decision may have been reasonable. The missing zone-failure test may still reveal weak information work. The late alert may reveal poor execution. The outage supplies evidence about all three without automatically collapsing them into one verdict.

So far, we have separated the choice from the result. This matters because each layer requires a different repair: gather better evidence, improve the decision rule, execute the plan more faithfully, or reduce exposure to unavoidable variance.

A Good Result Can Hide a Bad Decision

Consider a second team. It makes a full release with no canary because the change “looks small.” A load-test warning is dismissed. By chance, campaign traffic is much lower than forecast. Nothing fails.

The outcome is good. The decision process is weak.

The four possible combinations are:

This table is not a license to label every loser “strong.” The process needs evidence recorded before the outcome. Without that record, people can invent good reasons after the fact.

Check: A team skips a cheap security review, deploys, and suffers no breach. Was the deployment decision good because the outcome was good?

Think first, then reveal.

Answer: Not necessarily. The absence of a breach is a good outcome, but skipping relevant and inexpensive evidence can make the information process weak. One quiet period does not validate the choice.

Luck Is Not a Storage Box for Mistakes

“Luck” should mean variation that remained outside reasonable prediction or control after proportionate investigation. It should not mean “anything we did not bother to examine.”

Suppose the checkout team had an earlier report showing cache saturation during zone loss. If nobody read it, the outage was not purely bad luck. The team had an information-access failure.

Suppose the risk was documented, the failure mode was tested, and an independent provider defect violated its contract in a new way. Luck plays a larger role, though the outcome still teaches the team something.

A useful review asks three questions:

Was the information available?
Was it reasonably obtainable for this decision's stakes and deadline?
Would it have changed the choice or its controls?

This prevents two opposite errors:

• Hindsight punishment: “You should have known” when nobody could reasonably know.
• Ignorance protection: “We could not know” when the warning was cheap, relevant, and ignored.

Outcomes Are Evidence, Not Verdicts

Separating decision and outcome does not mean ignoring results. Outcomes update the model.

One cache overload does not prove that every staged rollout is wrong. It does show that the team's capacity model was incomplete under zone failure. Repeated overloads would place stronger pressure on the model and on any team that keeps making the same choice.

Use two review views:

• Ex ante review: Was the decision defensible before the result, using what was known or reasonably knowable?
• Ex post update: What does the result reveal about assumptions, probabilities, execution, and future controls?

The first view protects fair judgment. The second protects learning.

Check: A release decision was well documented and reasonable, but the same failure has now happened three times. Can the team keep calling each event bad luck?

Think first, then reveal.

Answer: No. Repeated outcomes are new evidence. A process that ignores them stops being reasonable because its information set and probability estimates should have changed.

Trade-offs and Limits

Process-based review improves learning, psychological safety, and the quality of future choices. People can report uncertainty without fearing that every unlucky result will be treated as incompetence.

It also costs time. A detailed decision record is wasteful for a cheap, reversible choice. The effort should grow with stakes, irreversibility, uncertainty, and cost of delay.

The central trade-off is between review discipline and decision speed. More evidence and a stronger record make later learning fairer, but excessive review can delay action until an opportunity disappears.

The distinction can also be abused. A polished memo can make a weak choice look respectable. Teams may call every bad result “variance” or redefine the original reasoning after the outcome.

The protection is contemporaneous evidence: write the important assumptions, confidence, risks, and review triggers before the result. Then compare the record with what happened.

This lesson does not yet tell us how to create a strong set of alternatives, assign probabilities, resolve value conflict, or choose between reversible and irreversible actions. Later lessons build those parts. Here, the essential move is simpler: preserve the boundary between the choice made under uncertainty and the result observed afterward.

Common Confusions

Confusion: A good decision is one that succeeds

Why it is tempting: success is visible and easy to measure.

Better model: success measures outcome quality. Decision quality asks whether the choice was defensible before uncertainty resolved.

Confusion: If the process was good, nobody is responsible

Why it is tempting: separating luck from judgment can sound like removing accountability.

Better model: responsibility becomes more precise. We can identify failures in evidence gathering, decision logic, execution, monitoring, or learning instead of blaming the final result alone.

Confusion: Bad luck explains every surprise

Why it is tempting: surprises were not in the preferred story.

Better model: first ask whether the risk was reasonably knowable, whether investigation was proportional, and whether prior outcomes should already have changed the model.

Practice: Review Before You Know the Ending

A company must choose whether to replace a stable but expensive data provider. A cheaper provider passes a small test, but the test does not cover peak traffic. The contract can be cancelled after one month, while a delayed migration costs €20,000. The team chooses a one-month parallel run with an automatic fallback. During week two, the cheaper provider has a three-hour outage.

Write a short review with four headings:

1. Information quality
2. Decision quality
3. Execution quality
4. Outcome quality and update

A good answer should:

• judge the choice using evidence available before the outage;
• distinguish the reversible parallel run from the provider's outcome;
• ask whether peak-load and outage evidence was reasonably obtainable;
• inspect whether the automatic fallback worked as planned;
• use the outage to update the next decision rather than to rewrite the original reasoning.

Connection to the Next Lesson

Once we stop asking only “Did it work?”, another problem becomes visible: teams often compare one favored plan with doing nothing. The next lesson, Options Before Optimization, shows why decision quality begins by creating real alternatives before scoring them.

Resources

• [BOOK] Thinking, Fast and Slow — Focus: how hindsight and outcome information distort judgment.
• [BOOK] How to Decide — Focus: separating decision process from result and recording reasoning before uncertainty resolves.
• [ARTICLE] Stanford Encyclopedia of Philosophy: Decision Theory — Focus: the formal relationship among acts, uncertain states, outcomes, and preferences.

Key Takeaways

• Decision quality is evaluated using what was known or reasonably knowable before the outcome.
• Information quality, decision quality, execution quality, and outcome quality require different questions and different repairs.
• A good outcome can reward a weak process, while a bad outcome can follow a defensible process.
• Luck does not excuse ignored, inexpensive, or repeatedly observed evidence.
• Outcomes are new evidence for the next decision, not the entire verdict on the previous one.