Feedback, Regret, and Decision Journals

By the end of this lesson, you will be able to...

• reconstruct what you believed, valued, and expected before a decision;

• separate outcome luck, decision quality, and execution when reviewing what happened;

• turn regret and feedback into one concrete update for the next decision.

Idea in one sentence: A decision journal preserves the before-state of a choice so that later feedback can improve the process without letting hindsight rewrite the evidence.

Core Insight

Consider the checkout team from the previous lessons. Before a high-traffic campaign, it chose a staged rollout with a canary, a degraded mode, and a stop condition. The team wrote down a 25% estimate for a serious checkout failure, named availability and customer trust as values, and ran a pre-mortem about cold caches and duplicate payments.

After the campaign, the result is mixed: no customer is charged twice, the canary stops expansion once, and the degraded path serves customers. However, one alert arrives late and the team spends an hour arguing about whether to continue. A simple story now becomes tempting:

“The rollout worked, so the decision was good.”

That sentence compresses several objects. The outcome may be good because the plan was sound, the defenses worked, luck spared an unimagined failure, or the team recovered well. A bad result can also follow a careful decision when an unlikely event occurs. Compare the record with the observed path, then update the model or control; memory stores the ending more vividly than the uncertainty that preceded it.

The Decision Journal as a Before-and-After Record

Write the first part before acting, while the choice is still uncertain. Keep it short enough that another person can inspect it.

1. Question and horizon. What choice is being made, by whom, and when will it be reviewed? “Can we expand the checkout canary?” is better than “Is the release safe?”
2. Context and evidence. What was known, unknown, and still being measured? Record the reference period, relevant incidents, and important missing data.
3. Options and baseline. List the current plan, a smaller experiment, a delay, and doing nothing. Include the learning path when it differs from the production path.
4. Belief and confidence. Define the event and give a probability or range. The checkout team might write, “There is a 20–30% chance that a zone loss causes a customer-visible checkout failure during the first hour.” Add what would make the estimate move.
5. Values and constraints. State what counts as a cost: payment integrity, availability, revenue, team fatigue, or customer trust. Mark veto constraints separately from preferences.
6. Commitment and optionality. Is this reversible? What is the undo cost, the cost of delay, and the information gained by a small step? Name the rollback authority.
7. Failure paths and controls. Carry forward the pre-mortem: assumption, signal, defense, owner, and stop trigger.
8. Review rule. Choose a date or event that will cause the review. Do not wait for a dramatic incident; routine outcomes provide better calibration data.

The after-state adds the actual path: what happened, which signals appeared, what action was taken, and which part of the plan produced that response. This connects the framing, belief, values, optionality, and failure controls from lessons 001–006.

Two-Pass Review: Protect the Ex Ante Record

Review in two passes. In the first pass, hide the outcome if possible. Read only the question, evidence, options, probabilities, values, and controls. Ask:

• Was the event defined clearly enough to forecast, with alternatives and a baseline visible?
• Did the probability match the evidence available then?
• Did values, reversibility, and hard constraints change the choice explicitly?
• Could a leading indicator have triggered a different action?

This is an ex ante judgment: was the process defensible at the time? It is not a claim that the option must win.

In the second pass, reveal the outcome and trace the path. Compare forecast with observation, but do not treat the difference as a score by itself. Classify the gap:

The classification is a hypothesis, not a verdict. More than one gap can be present. A journal earns its space when it changes a future estimate, trigger, owner, or option.

Check: The rollout avoided errors, but only because traffic stayed low. Was the original probability estimate proven wrong?

Think first, then reveal.

Answer: Not necessarily. The observed case may contain little information about the high-traffic event. Record the favorable outcome, but also record that the stress condition was not tested. Update confidence in the evidence, not just the headline result.

Regret Is a Signal, Not a Verdict

Regret often arrives as one sentence: “We should have waited,” or “We should have expanded sooner.” Split it before acting on it.

• Outcome regret says the result was painful: customers saw errors, revenue fell, or the campaign missed its target. It points to a cost, but not automatically to a bad decision.
• Process regret says the team violated a defensible rule: it ignored a stop condition, hid uncertainty, or failed to name a rollback owner.
• Counterfactual hindsight imagines that the better path was obvious because the ending is now known. It is a warning that the before-state has been lost.

Suppose the checkout team feels regret because the canary stopped expansion and the campaign served fewer customers. That is outcome regret. If the team stopped exactly when the pre-mortem trigger fired, the process may be good even though the business result is disappointing. If it continued past the trigger because no owner wanted to make the call, that is process regret and a clear control failure.

If the team ignored the rehearsal and happened to see no errors, the outcome is good but the process is weak. Luck does not validate the missing control.

A Worked Journal Review

Here is a compact record for the checkout campaign.

Before the decision

• Question: Should we move from 10% to 25% traffic after the canary?
• Options: stay at 10% and collect data; move to 25% with rollback; delay the campaign; expand without a stop rule.
• Belief: 20–30% chance of customer-visible checkout failure in the first hour at 25% traffic.
• Values and constraint: protect payment integrity first; tolerate a temporary feature reduction; never allow duplicate payment requests.
• Commitment: use the reversible 25% step; roll back if checkout errors exceed 1% for five minutes or if duplicate-key checks fail.
• Pre-mortem control: the incident owner watches payment-path errors and queue depth; the platform owner confirms warm-cache recovery.

Observed path

The canary showed rising queue depth, so the incident owner stopped expansion. The degraded path preserved payment integrity. Later, an alert was acknowledged after nine minutes because the owner was also coordinating the campaign. Traffic never reached the stress level needed to test the cold-cache scenario fully.

Review

The decision process was defensible: it preserved an option, stated a constraint, and used a trigger. The alert ownership and escalation path were weak. The favorable outcome is partly due to the stop condition and partly due to the absence of the worst load. The next update is not “be more confident.” It is: assign a separate alert owner, rehearse the high-load path, and keep the probability range until that evidence exists.

Keep the estimate as a range until the stress test supplies evidence; change the control without pretending the original choice was flawless or foolish.

Feedback Must Close a Loop

Feedback becomes useful when it changes a future action:

record before -> act -> observe path and outcome -> review without hindsight -> update model or control -> record next decision

An update can target different layers:

• Model: revise an event definition, reference class, base rate, or probability range.
• Evidence plan: run a test that was missing, or measure a leading indicator earlier.
• Decision rule: change a threshold, rollback condition, or review date.
• Execution control: assign an owner, rehearse a path, or improve escalation.
• Values and framing: add a stakeholder cost or constraint that was invisible before.

Do not update all layers after every result. A small change is easier to test than “the whole approach failed.”

Check: A decision produced a bad result, but the team followed its stated process and the event was in the low-probability tail. What should the journal record?

Think first, then reveal.

Answer: Record the loss, the process evidence, and the possibility of bad luck. Then ask whether the impact justified a stronger defense or a different probability range. Do not rewrite the process as irrational merely because the tail occurred.

Trade-offs, Limits, and Common Confusions

The central trade-off is between review discipline and attention. A short journal preserves evidence; a long form becomes bureaucracy. It can also become self-defense that explains why the author was right instead of exposing what would change their mind.

Keep a field when it affects the choice, the review, or the next control. Omit decorative history.

Common confusions:

1. “Good outcome means good decision.” Separate process, execution, luck, and result.
2. “A bad outcome proves the probability was wrong.” One draw can fit a reasonable range.
3. “Regret tells us what we should have done.” Classify outcome regret, process regret, and hindsight first.

Practice: Review One Real Decision

Choose a recent uncertain choice without exposing private details. Write the before-state and observed path separately, then complete this review:

1. What was the most defensible part of the process?
2. What was luck, execution, or a model gap?
3. Which regret is present, and what kind?
4. What one probability, trigger, owner, value, or option will change next time?
5. When will you review the next case?

If you cannot state an update, the review has become storytelling. Return to the decision question and make the next change concrete.

Connection to the Capstone

The track has now built a complete review vocabulary: separate decision quality from outcome, generate options, state calibrated beliefs, expose values, preserve optionality, and turn failure imagination into controls. The next lesson uses those pieces before the result is known. You will write a decision memo that another person can challenge, execute, and review later.

Resources

• [BOOK] Thinking, Fast and Slow — Focus: memory, hindsight, and the pressures that distort judgment.
• [BOOK] How to Decide — Focus: separating decision quality from outcomes and using feedback to improve choices.
• [ARTICLE] Stanford Encyclopedia of Philosophy: Decision Theory — Focus: actions, uncertainty, consequences, and the limits of retrospective evaluation.

Key Takeaways

• A decision journal preserves the context, alternatives, beliefs, values, controls, and review rule that existed before the result.
• Two-pass review protects ex ante judgment: inspect the process first, then explain the outcome using luck, model, execution, and framing hypotheses.
• Outcome regret and process regret carry different information; hindsight is a reason to return to the original record.
• Feedback is complete only when it changes a future estimate, evidence plan, decision rule, execution control, or value constraint.