Foundations Review Check

By the end of this lesson, you will be able to...

• reduce a distributed-systems scenario to promise, evidence, uncertainty, mechanism, and trade-off.

• choose mechanisms by naming the failure pressure they address.

• review a workflow for retries, stale reads, ownership, overload, contracts, observability, and degraded behavior.

Idea in one sentence: A good distributed-systems review starts with the promise at risk, then uses evidence to choose the smallest mechanism that makes safe progress under uncertainty.

Core Insight

A team uses a collaboration app for project work.

Mina creates a workspace called launch-plan and invites Lee. Mina sees the workspace immediately. Lee opens the invitation link and gets:

Workspace not found.

Mina clicks "send invite" again. Lee receives two emails. A few minutes later, the notifications queue is growing. An old notification worker starts rejecting a new member_invited event because it does not understand the new invite_role field.

This is not one bug. It is a review exercise.

Several questions are tangled together:

• Did workspace creation commit?
• Which service owns the official membership state?
• Is Lee reading from a stale replica?
• Did Mina's retry create a duplicate invite?
• Is the notification queue overloaded?
• Did a message contract change break an old consumer?
• What should the app tell Lee while evidence is incomplete?

A weak review says:

Just retry.
Add a queue.
Use stronger consistency.
Make a dashboard.

A strong review says:

promise:
  one accepted invite should give Lee access to one workspace

evidence:
  Mina has a workspace id; Lee has an invite link; notification queue is growing

uncertainty:
  membership may be committed, stale, duplicated, delayed, or rejected by an old consumer

mechanism:
  stable invite operation id, membership owner, read-your-writes status path,
  bounded notification queue, compatible event rollout, degraded pending state

trade-off:
  slightly slower confirmation and more explicit pending states in exchange for
  avoiding duplicate invites and misleading "not found" errors

The point of this review lesson is not to memorize every term again. It is to practice selecting the right concept when a scenario contains several plausible failures.

The Review Frame

Use this five-line frame before proposing a fix:

promise:
evidence:
uncertainty:
mechanism:
trade-off:

The frame slows you down in a useful way.

The promise is what must remain true for the user or operator.

The evidence is what the system can actually prove from records, messages, traces, metrics, and local observations.

The uncertainty is what the system does not yet know. A timeout, stale read, missing event, or queue delay should not be treated as proof of the opposite outcome.

The mechanism is the design or operational control that changes system behavior.

The trade-off is the cost: latency, availability, complexity, stale data, manual repair, compatibility code, or a smaller user promise.

Check: Lee sees Workspace not found after Mina created the workspace. What is the most careful first interpretation?

Think first, then reveal.

Answer: The app has conflicting evidence. Mina has some local evidence that creation succeeded, while Lee's read path did not find it. That could be stale replication, missing membership, a bad invite, delayed event processing, or a real failed create. The review should not jump straight to one cause.

Worked Review: The Invite Incident

Follow the incident step by step.

1. Input: One User Promise

The user promise is narrow:

When Mina invites Lee to launch-plan,
Lee should either get access to that workspace
or see a truthful pending/unavailable state.
Lee should not receive duplicate invites for one intent.

This promise is stronger than "an email was sent." Email is only one side effect. The real promise includes workspace existence, membership, invite identity, and a user-visible state.

2. Transition: Separate Local Evidence

Each component sees only part of the story.

workspace service:
  workspace_id=ws-42 created by Mina

membership service:
  maybe has invite invite:ws-42:lee

notification service:
  queued two emails

Lee's read path:
  workspace not found

event consumer:
  rejected member_invited with unknown invite_role

Intermediate state:

creation may be durable
membership may be missing or delayed
notification side effect may be duplicated
read path may be stale
event contract may be broken for one consumer

The system has evidence, but not yet one joined truth.

3. Decision: Pick Mechanisms By Pressure

Now choose mechanisms because of the pressure they address.

Duplicate invite pressure:

use invite_operation_id = invite:ws-42:lee
make the receiver idempotent for that operation id

Stale read pressure:

route Lee's first access check to the membership owner
or carry a version token so the read path does not move backward

Overload pressure:

bound notification queue age
rate-limit repeated invite sends
shed optional notification enrichment

Contract pressure:

deploy consumers that tolerate invite_role before producers depend on it
track rejected messages and replay old versions safely

Uncertain user state:

show invite_pending instead of workspace_not_found
when creation evidence exists but membership evidence has not arrived

Output:

The review answer becomes a small plan: preserve one invite intent, find the authoritative membership state, avoid duplicate notifications, stop queue amplification, and repair or replay rejected events.

Naive failure contrast:

If the system "just retries," it may send more duplicate emails. If it "just adds a queue," it may hide a growing backlog. If it "just uses stronger consistency everywhere," it may buy cost and latency where a targeted owner read would have solved the confusing state.

Coverage Map

Use this map to connect symptoms to track concepts.

timeout or missing response
  -> what does the caller know, and what remains unknown?

duplicate user action
  -> what operation id makes retries one logical intent?

two views disagree
  -> which read path may be stale, and who owns the official fact?

one decision must not fork
  -> does this need a single owner, quorum, or consensus-like coordination?

events arrive in a surprising order
  -> what causal dependency was assumed?

queue age grows
  -> where should admission, backpressure, deadlines, or shedding apply?

new event breaks old worker
  -> which schema versions and meanings must coexist?

operators cannot explain one case
  -> what logs, traces, metrics, and durable records fail to join?

normal promise becomes unsafe
  -> what smaller degraded promise remains honest?

The review skill is choosing the line that matches the pressure. Not every scenario needs every mechanism.

Common Confusions

Confusion: Mechanism First, Promise Later

Why it is tempting:

Tool names are easy to reach for. They sound decisive.

Better model:

Mechanisms earn their place by protecting a named promise. "Use a queue" is incomplete until you say what waits, how long it may wait, and what happens when waiting becomes harmful.

Confusion: Unknown Means Failed

Why it is tempting:

When a caller times out or a read returns empty, the UI needs to show something.

Better model:

Unknown is a real state. If the system cannot prove success or failure yet, it should expose a truthful pending, retryable, or unavailable state and keep the evidence needed for repair.

Confusion: Review Means Finding One Root Cause

Why it is tempting:

Incident stories often want one neat cause.

Better model:

Distributed failures often contain several pressures at once: stale reads, retries, overload, contracts, and missing evidence. A good review separates them and chooses the next safe action for each.

Practice Drill 1: Classify The Pressure

For each symptom, name the main pressure and one mechanism to inspect.

A. A user clicks "send invite" twice after the first request times out.
B. A dashboard in region 1 shows membership, region 2 says no access.
C. A queue has 20,000 notification jobs and oldest age is 40 minutes.
D. A new event field causes one old worker to dead-letter messages.
E. Support cannot answer whether Lee was ever granted access.

Model answer:

A. retry/duplicate side effect -> stable operation id and idempotent receiver
B. stale read/ownership -> read policy, owner routing, version token
C. overload/backpressure -> admission control, queue age limit, shedding
D. contract evolution -> forward/backward compatibility, rollout order, replay path
E. observability/evidence -> joinable ids across logs, traces, events, records

Practice Drill 2: Write The Five-Line Review

Use the invite incident and write:

promise:
evidence:
uncertainty:
mechanism:
trade-off:

Model answer:

promise:
  one invite intent gives Lee one clear access outcome

evidence:
  ws-42 was created; invite emails were queued twice; one consumer rejected member_invited

uncertainty:
  membership may be delayed, duplicated, stale on Lee's read path, or blocked by contract failure

mechanism:
  idempotent invite_operation_id, owner-backed access check, bounded queue,
  compatible event rollout, invite_pending state, repair evidence

trade-off:
  more explicit intermediate states and compatibility code in exchange for
  fewer duplicate side effects and clearer repair

Check: Which part of the five-line review prevents "just add a queue" from being accepted as a complete answer?

Think first, then reveal.

Answer: The promise and trade-off lines. They force the mechanism to explain what it protects and what it costs. A queue can help delivery, but without deadlines and backpressure it may only hide stale work.

Readiness Check For The Capstone

You are ready for the capstone when you can do three things.

First, keep evidence narrow:

A timeout means the caller stopped waiting.
An empty read means this read path did not find the record.
A queued message means work is intended, not completed.

Second, name the owner of official facts:

who owns workspace creation?
who owns membership?
who owns notification side effects?
who owns recovery after contract failure?

Third, connect every mechanism to a failure and cost:

idempotency -> duplicate attempts, at the cost of operation identity storage
owner read -> stale reads, at the cost of latency or availability during partition
backpressure -> overload, at the cost of rejecting or delaying work
compatibility -> safe deploys, at the cost of temporary old-version support
degraded mode -> unsafe normal promise, at the cost of a smaller user experience

The capstone will ask for a complete design. This review asks whether the parts of your reasoning are sharp enough to combine.

Review Rubric

Use this small rubric on your own answer.

Strong answers:

• name one user-visible promise;
• separate local evidence from missing evidence;
• avoid treating unknown as failed;
• choose one mechanism because it changes a specific failure path;
• name the cost of that mechanism;
• include the evidence needed for repair or recovery.

Weak answers:

• start with a tool name;
• collapse several failures into one vague cause;
• assume retries are safe without operation identity;
• hide stale reads behind a generic dashboard problem;
• ignore old message versions or replay;
• promise normal behavior when degraded behavior would be more honest.

The rubric is intentionally small. It should fit in your head while reading the capstone scenario.

Use it whenever a scenario feels noisy or vague.

Resources

• [ARTICLE] Notes on Distributed Systems for Young Bloods - Focus: Practical failure framing and the habit of respecting uncertainty.
• [BOOK] Designing Data-Intensive Applications - Focus: Replication, consistency, encoding, partitioning, and system trade-offs.
• [BOOK] Site Reliability Engineering: Monitoring Distributed Systems - Focus: Connecting user-visible promises to signals and evidence.

Key Takeaways

• Start review answers with promises and evidence, not tool names.
• Treat timeout, empty read, queue delay, and rejected message as clues, not full conclusions.
• Choose mechanisms by naming the pressure they address and the trade-off they introduce.
• A strong review separates duplicate side effects, stale reads, overload, contract drift, missing evidence, and unsafe normal promises.
• The capstone combines the same review frame into a complete architecture decision.