Pre-Mortems and Failure Imagination

By the end of this lesson, you will be able to...

• Turn a chosen plan into a specific future failure story with a visible causal path.

• Connect failure paths to leading indicators, defenses, owners, and trigger conditions.

• Distinguish a useful pre-mortem from a list of frightening possibilities that never changes the plan.

Idea in one sentence: Imagine the plan has already failed, then use the story to expose assumptions and add a defense before the real event.

Core Insight

Consider the checkout team after choosing warm failover and a limited degraded mode. The canary is scheduled for Friday. Before it starts, the incident lead asks everyone to imagine that Monday's review headline says:

The campaign failed because checkout became unavailable again.

The team is not being asked to predict the future. It is being asked to work backward from a concrete failure and find the path that could produce it.

One engineer says, “The new zone never had enough capacity.” Another says, “The fallback worked, but retries duplicated payment requests.” A third says, “The alert fired, but nobody owned the decision to stop expansion.”

These stories are useful because each exposes a different assumption, signal, or defense. A vague warning such as “something might go wrong” does not.

The Naive Exercise: List Everything Scary

Teams often handle risk with a long list:

• provider outage;
• bad deploy;
• overload;
• security problem;
• human error;
• unknown unknown.

The list feels responsible. It rarely tells anyone what to do next.

The opposite mistake is to discuss only the expected success path. That makes the plan easy to approve and hard to operate when the first assumption breaks.

A pre-mortem sits between these extremes. It begins with a selected plan, assumes that the plan has failed at a defined future time, and traces a small number of plausible paths from cause to consequence.

The Mechanism Step by Step

Step 1: Freeze the plan and the time horizon

Write down the commitment being reviewed:

Warm failover is serving 25% of campaign traffic, with degraded checkout available when capacity or recovery thresholds are crossed.

Choose a horizon:

It is Monday morning, after the first high-traffic weekend.

The time horizon matters. A plan can be safe for the first hour and fail after configuration drift, staff rotation, or a slow data buildup.

Step 2: Assume a meaningful failure

Do not ask for every imaginable disaster. Choose a failure that would change the decision or the controls:

Checkout was unavailable for twelve minutes, or a payment-integrity boundary was violated.

The failure should be specific enough to investigate and important enough to justify attention.

Step 3: Tell one causal story

Use this chain:

trigger -> hidden condition -> mechanism -> user impact -> visible signal

For example:

zone loss
-> warm capacity was not actually warmed
-> requests concentrated on one cold cache
-> checkout latency and errors rose
-> saturation and queue depth crossed the stop threshold

The chain turns fear into a mechanism. It also shows where a defense could interrupt the path.

Step 4: Name the assumption under pressure

The story depends on an assumption:

“Warm capacity will accept traffic at the expected rate when one zone disappears.”

If the team cannot state the assumption, it cannot test or monitor it.

Step 5: Add a defense, owner, and trigger

The response must change the plan. It might be a test, a design guard, a runbook step, or a rollback condition.

Run a zone-loss rehearsal before expanding past 25%. The SRE on call owns the result. If recovery exceeds 30 seconds, stop expansion and keep degraded checkout active.

The pre-mortem is complete when the story produces an action or a deliberate acceptance of the risk.

A Worked Failure Trace

The team writes four failure stories for the same rollout.

The table is not a catalogue of generic risks. Each row connects a story to an assumption, a signal, and a defense.

Now trace one row in detail.

1. Input: a cloud zone loses connectivity during the campaign.
2. Transition: traffic routes to the reserved zone.
3. Intermediate state: the cache is cold, so many requests reach the origin together.
4. Output: queue depth and checkout errors rise; the degraded mode opens.
5. Naive failure: the team watches average latency, which stays acceptable while a smaller payment path is already failing.

The better defense is not “monitor performance.” It is to watch the payment-path error rate, cache-fill pressure, queue depth, and recovery time that reveal this particular path.

So far, the pre-mortem has made an invisible chain inspectable. It has also created an early decision: if the warm-up rehearsal fails, expansion stops before customers discover the same weakness.

Check: A pre-mortem says, “The system might be attacked.” What should the facilitator ask next?

Think first, then reveal.

Answer: Ask for a specific attack path, affected assumption, observable leading signal, and defense. Without those, the sentence cannot change the plan.

Leading Indicators Are Earlier Than Outcomes

An outcome tells the team that the plan has already crossed a boundary. A leading indicator gives it a chance to act earlier.

For the checkout rollout:

• Outcome: checkout errors exceed 1% for five minutes.
• Leading indicator: cache warm-up misses the expected fill rate, or the payment queue grows for three consecutive intervals.

Leading indicators are not magic predictors. They are signals connected to a failure mechanism. A noisy metric that never changes a decision is not a useful indicator.

For each indicator, define:

1. what it measures;
2. which failure path it represents;
3. the threshold or trend that matters;
4. who decides what happens next.

This is why a pre-mortem belongs beside a plan, not in a separate risk document that nobody reads during the rollout.

Failure Imagination Is Not Probability Forecasting

The previous lesson assigned probabilities to defined events. A pre-mortem serves a different purpose.

• Probability asks: how likely is this event under the current model?
• Pre-mortem asks: if this event happened, what path could produce it and what would we change now?

A low-probability failure can deserve attention when its impact is severe and a cheap defense exists. A high-probability, low-impact failure may need a simpler control.

Do not use a pre-mortem to claim that an imagined story will happen. Use it to make assumptions and defenses visible.

Check: The team imagines ten unrelated disasters, but adds no tests, thresholds, or owners. Has it completed a useful pre-mortem?

Think first, then reveal.

Answer: No. It has produced pessimism without a control loop. A useful pre-mortem converts a small set of plausible stories into actions, signals, owners, or explicit risk acceptance.

What This Changes

Before this lesson, the team might ask:

Does everyone feel confident about the rollout?

After this lesson, it can ask:

Which failure path would make this plan wrong, what would we see first, and who has authority to change course?

The question is operational. It turns confidence into a set of observable conditions.

It also protects disagreement. An engineer who worries about cold caches does not need to win the whole argument. The concern can become a rehearsal, signal, owner, and stop condition. If the test passes, the concern is reduced. If it fails, the plan changes before the campaign.

Trade-offs and Limits

The central trade-off is between failure coverage and attention. More scenarios can expose more assumptions, but a long list can bury the few paths that deserve action.

Pre-mortems cost time and can create anxiety. They can also become pessimism theater: people compete to invent dramatic failures while avoiding the ordinary failure that the system can actually show.

The technique does not guarantee that the team imagines the right path. It does not replace probability estimates, testing, monitoring, or a capable response. It can fail when participants are afraid to name an assumption, when owners cannot act, or when thresholds are too vague to trigger a decision.

Use a stopping rule: keep a failure story when it is plausible, material, and connected to a test, signal, defense, owner, or explicit acceptance. Drop it when it is only a dramatic variation that cannot change the plan.

Common Confusions

Confusion: A pre-mortem predicts the actual failure

Why it is tempting: the exercise uses a future failure story.

Better model: it is a structured way to expose assumptions and defenses. The story is a test of the plan, not a prophecy.

Confusion: Every imaginable failure deserves a control

Why it is tempting: ignoring a risk feels irresponsible.

Better model: prioritize plausible, consequential paths where a signal or defense can change the outcome.

Confusion: A leading indicator is just an early metric

Why it is tempting: any metric that moves before an outage looks predictive.

Better model: the indicator must connect to a failure mechanism and a decision owner. Otherwise it is noise with a timestamp.

Confusion: Naming a defense means the risk is solved

Why it is tempting: a checklist feels like control.

Better model: a defense needs evidence that it works, a trigger, and an owner who will act under pressure.

Practice: Pre-Mortem the Event-Store Migration

Return to the event-store migration. The team is considering a parallel run with automatic fallback, but regional recovery is still unknown.

Write a pre-mortem set six weeks after the migration. Create three failure stories. For each one, record:

1. the trigger;
2. the hidden assumption;
3. the mechanism and user or business impact;
4. one leading indicator;
5. one defense or experiment;
6. the owner and stop or restore condition.

At least one story should involve data integrity, one should involve operations or recovery, and one should involve the cost of waiting or the degraded experience.

A good answer should trace a causal path instead of listing nouns such as “latency” or “security.” It should change the migration plan or state clearly why the team accepts the remaining risk.

Connection to the Next Lesson

The team now has a decision record, forecasts, values, commitment levels, failure paths, and controls. The next lesson, Feedback, Regret, and Decision Journals, will show how to record the decision before the outcome and learn from it without rewriting the past.

Resources

• [BOOK] Thinking, Fast and Slow — Focus: availability, overconfidence, and why vivid failure stories can distort judgment.
• [BOOK] How to Decide — Focus: pre-mortems, decision process, and turning imagined failure into a better choice.
• [ARTICLE] Stanford Encyclopedia of Philosophy: Decision Theory — Focus: uncertainty, consequences, and the limits of reasoning from imagined states.

Key Takeaways

• A useful pre-mortem starts from a chosen plan, assumes a specific future failure, and traces a causal path backward.
• Each important failure story should expose an assumption and produce a signal, defense, owner, or explicit risk acceptance.
• Leading indicators matter because they create a chance to act before the user-visible outcome.
• Failure imagination is not probability forecasting and does not replace testing or monitoring.
• The central trade-off is failure coverage versus attention; a long list is not a control loop.