Postmortems and Decision Records

By the end of this lesson, you will be able to...

• Separate evidence, inference, impact, and follow-up in a short incident reconstruction.

• Write a decision record that preserves context, alternatives, consequences, and a revisit condition.

• Diagnose when a document is blaming a person, rewriting history, or hiding a trade-off.

Idea in one sentence: A postmortem explains what happened and what must improve; a decision record explains why the team chose one path while alternatives were still open.

Core Insight

The import queue incident from lesson 009 is over. Queue age returned to normal after the team paused intake and database on-call increased the connection-pool limit. The immediate pressure is gone, but the writing work has just changed shape.

Someone writes, “Nora forgot to increase the database pool after the deployment, so we increased it.” This sentence is short. It is also a poor record. It mixes a claim about a person, a possible cause, and a later decision. A reader cannot tell what happened, what the evidence shows, which alternatives were considered, or how the same failure will be less likely next time.

Two documents solve different problems here:

• A postmortem reconstructs a past event so a future reader can learn from its impact, timeline, evidence, system conditions, and follow-up work.
• A decision record preserves a choice while choices are still open: the context, alternatives, decision, consequences, owner, and condition for revisiting it.

Both documents build trust through inspectable reasoning. Neither is a place to make the writer look certain after the fact.

The Documents Make Different Promises

The topic may be the same import service. The reader's job is different.

The naive design is one large “incident notes” page. It begins with a timeline, adds speculative blame, records a later design choice, and leaves a list of wishes such as “improve monitoring.” The page has information, but no reader can rely on its boundaries.

Plain meaning:

Write the past as a reconstruction. Write the choice as a decision. Link the two when the past created the pressure for the choice.

In our import situation:

The postmortem can say database saturation contributed to the backlog. A later decision record can say the team chose a lower concurrency cap because it protects the shared pool, while accepting slower peak import throughput.

Technical name:

These are accountability artifacts. They make claims, uncertainty, ownership, and consequences visible enough for a future reader to inspect rather than reconstruct from memory.

A Small Evidence Discipline

An incident document becomes unreliable when it gives every sentence the same force. Separate what the team observed from what it inferred and from what it decided to do.

The word likely is not a weak substitute for thinking. It tells the reader where the evidence stops. If later logs show a different cause, the postmortem can be corrected without pretending the first conclusion was certain.

A Worked Incident Reconstruction

Here is a compact postmortem slice for the queue backlog.

Summary and impact

On 2026-07-20, new Parcel API imports remained queued for up to 34 minutes. Completed imports and downloads remained available. The incident affected new import processing from 09:07 to 09:41.

Timeline

What we know and what we infer

Evidence: The deployment doubled concurrent workers. Database saturation rose above 90%. Claims fell. When intake was paused and the pool limit was raised, claims recovered and queue age fell.

Inference: The higher concurrency likely exceeded the connection pool's comfortable operating range. The deployment was a trigger. The pool limit and missing saturation guard were system conditions that allowed one change to affect import processing.

Not established: We have not shown that concurrency 16 would fail under every traffic pattern, or that increasing the pool is harmless for every database workload. Those are questions for measurement and a separate decision.

Follow-up work

This reconstruction has input, intermediate evidence, a response, and a recovery decision. It also makes the naive failure visible: “Nora caused the incident” compresses a system of conditions into blame and teaches nothing useful about prevention.

So far, the postmortem gives future readers an honest past. It does not yet answer what the team should permanently change.

Check: Which sentence belongs in the postmortem as an inference, rather than as an established fact?

Think first, then reveal.

Answer: “Higher concurrency likely exhausted the connection pool under this traffic pattern.” It is supported by the timeline but still needs measurement. “Database saturation was 94% at 09:12” is evidence. “Cap concurrency at eight” is a future decision.

The Decision Record: Preserve the Choice

An architectural or technical decision record should not replay the whole incident. Its reader already needs the current choice and the reasoning that made it preferable at the time.

Decision: cap import-worker concurrency at eight

Context. The incident showed that concurrency 16 can saturate the shared database pool under observed traffic. The team needs a safer default before the next import-heavy customer launch.

Alternatives considered.

Decision. Use a default concurrency cap of 8 for all import workers. Keep the current database pool limit. Run two load tests before 2026-08-05 to measure queue age, pool saturation, and throughput. Priya owns the configuration; Nora owns the measurements.

Consequences. The team accepts a slower peak import rate in exchange for lower immediate risk to the database. Support should not promise prior peak processing time during this period. The cap may be raised only through a new decision after the measurements are reviewed.

Revisit condition. Revisit this decision if pool saturation remains below 70% while queue age exceeds 15 minutes during a representative load test, or if a customer contract requires a throughput target the cap cannot meet.

The record has a visible before-and-after decision boundary. It does not say that eight is universally correct. It states what was known, what was chosen, and what evidence can reopen the choice.

Check: Why should the record name the rejected “increase the pool now” alternative?

Think first, then reveal.

Answer: A future reader can see that the team considered throughput but deferred it because of uncertainty about other workloads. Without the alternative and its cost, the cap can look arbitrary and the same discussion will return from zero.

Common Confusions

Confusion: Blameless means vague

Why it is tempting:

Removing names can feel like removing accountability.

Better model:

Be precise about actions, information, system conditions, ownership, and follow-up. Avoid turning a person's name into a causal explanation when the document needs a mechanism the system can improve.

Confusion: A timeline proves causation

Why it is tempting:

An event before a failure feels like a complete explanation.

Better model:

Timeline order provides evidence to investigate. State the additional signals, tests, or uncertainty that support the causal claim.

Confusion: A decision record must be updated until it describes current reality

Why it is tempting:

Readers want the latest answer in one place.

Better model:

Keep an accepted decision as historical context. When the choice changes, create a new record that supersedes it and link the two. Otherwise the reason for the old decision disappears.

Trade-offs and Limits

These documents improve organizational memory, learning, and accountability. The trade-off is time and discipline: gathering evidence, writing calibrated claims, assigning one owner, and reviewing follow-up work cost effort after an already tiring event.

They do not guarantee a root cause, prevent every recurrence, or settle a technical choice forever. The boundary appears when evidence is incomplete, legal or privacy constraints limit sharing, or the situation changes after the decision. Say what is unknown, redact responsibly, and create a new decision record instead of silently rewriting history.

The useful signal is future use. If a new teammate can explain what happened without blaming someone, find the action owner, and understand why the current cap exists, the artifacts are doing their work.

Practice

Choose a small incident or a fictional one: a deploy caused elevated errors, a batch job overran, or a storage bucket became unavailable. Write two linked artifacts.

First, create a postmortem slice with impact, three timed evidence entries, one inference, one uncertainty, and two owned action items. Then write a decision record with context, two alternatives, one decision, a consequence, and a revisit condition.

A good answer should:

• label evidence differently from inference and decision;
• explain system conditions without turning a person into the cause;
• give each action a single owner and observable completion state;
• preserve a rejected alternative and its trade-off; and
• state what future evidence would reopen the decision.

Resources

• [BOOK/TUTORIAL] Google SRE Workbook: Postmortem Culture — Focus: Study clear timelines, blameless language, concrete owned action items, and follow-up that prevents recurrence.
• [REFERENCE] AWS Prescriptive Guidance: Architectural decision record process — Focus: Compare context, decision, consequences, ownership, state, and supersession in a durable decision log.
• [ARTICLE] Google SRE: Incident management guide — Focus: Notice how timely, reviewed postmortems turn incident evidence into prioritized corrective work.

Key Takeaways

• A postmortem reconstructs a past event; a decision record preserves a choice made under stated conditions.
• Evidence, inference, impact, decision, and follow-up should be visibly different kinds of statement.
• Blameless writing is precise about system conditions and ownership without using a person as the explanation.
• A good decision record includes rejected alternatives, consequences, and a condition that can reopen the choice.
• These artifacts stay useful when follow-ups are owned, measurable, reviewed, and linked to the evidence that created them.