@jepemo
EXIT

Foundations Review and Capstone Synthesis

Learning / Distributed Systems / Distributed Systems Foundations / 016

LESSON

Distributed Systems Foundations

016 50 min beginner CAPSTONE

Foundations Review and Capstone Synthesis

By the end of this lesson, you will be able to...

  • turn a distributed workflow into a short architecture decision record.

  • connect ownership, retries, consistency, overload, contracts, degraded modes, and evidence in one design.

  • review a capstone answer for honest intermediate states, named trade-offs, and recovery signals.

Idea in one sentence: A reviewable distributed design starts from the promise it must protect, then names the mechanisms and evidence that make the promise believable when the system is uncertain.

Core Insight

A concert venue sells 50,000 tickets for a popular event.

The product promise sounds simple:

If a customer buys one seat,
the system should not sell that same seat to someone else,
should not charge the customer twice,
and should give operators enough evidence to repair unclear cases.

That promise crosses many boundaries.

browser
  -> edge API
  -> seat inventory owner
  -> payment service
  -> ticket issuer
  -> email and wallet delivery
  -> observability and incident tools

Now add real distributed pressure.

The customer clicks "buy." The payment provider times out. The browser retries. The seat inventory service is in another region. A notification queue grows. A new ticket_issued event contains a field old wallet workers do not understand. During the incident, support asks whether one specific customer owns seat A-14.

No single concept from this track solves that whole story.

Retries help with transient failure, but unsafe retries can duplicate a charge. Replication helps reads, but a stale read can show the wrong seat status. Queues decouple work, but an old queue can hide stale promises. A schema version helps deploys, but not if old consumers reject retained messages. A dashboard can show pressure, but only durable evidence can explain one ticket.

The capstone move is to write a small architecture decision record, or ADR.

Plain meaning:

An ADR is a short design note that says what promise the system protects, what decision it makes, what evidence it keeps, and what trade-offs it accepts.

In this scenario:

The ADR should explain how the ticketing workflow prevents double sale and duplicate charge while still giving users honest states during payment, inventory, overload, and recovery uncertainty.

Technical name:

This is a capstone synthesis: a design artifact that combines the track's mental models into one reviewable decision.

The Capstone Scenario

Design the ticket purchase workflow for one event.

The system must support:

customers:
  search seats
  reserve a selected seat briefly
  pay for one reservation
  receive a QR ticket or a clear pending/unavailable state

operators:
  close admission during overload
  repair pending purchases
  explain one customer's path after an incident
  replay ticket events safely after a deploy

The expected failures are:

payment provider timeout
browser retry after timeout
seat inventory region becomes unreachable
notification queue grows during a sale spike
old wallet worker rejects a new ticket event field
support asks whether customer C owns seat A-14

Your job is not to design an entire commerce platform. Your job is to make one narrow workflow reviewable.

Check: Why is "use a queue and retry payment" not enough for this capstone?

Think first, then reveal.

Answer: It names tools without naming the protected promise, the operation identity, the owner of the seat, the honest state while payment is unknown, the overload policy, the message compatibility plan, or the evidence needed for repair.

ADR Shape

Use this structure:

# ADR: Ticket Purchase Under Partial Failure

## Context
## Decision
## Consequences
## Evidence To Retain
## Open Risks

The ADR should answer five review questions:

promise:
  What must remain true for the customer and venue?

ownership:
  Which service owns the final seat and ticket facts?

uncertainty:
  What states appear when payment, inventory, or delivery evidence is incomplete?

mechanisms:
  Which mechanisms protect the promise under retries, overload, partitions, and deploys?

evidence:
  Which ids, records, events, metrics, and repair states let operators reconstruct one case?

This structure keeps the design from becoming a shopping list of tools.

Worked ADR: Ticket Purchase Under Partial Failure

The following is a model answer. It is intentionally narrow. It does not claim to be the only valid design.

Context

The event has assigned seats. The strongest promise is:

One confirmed seat can belong to at most one customer,
and one customer purchase intent can create at most one successful charge.

Seat ownership and payment ownership are different facts.

seat_id:           event-9/A-14
reservation_id:    hold-82
purchase_intent:   buy:event-9/A-14/customer-7
payment_operation: pay:hold-82
ticket_id:         ticket-501

The seat inventory owner decides whether seat A-14 is held, released, or sold. The payment service records payment outcomes by payment_operation. The ticket issuer creates a QR ticket only after it sees both an owned seat decision and payment evidence.

The important uncertainty is that a timeout is not a business outcome. If the payment call times out, the system does not know whether the provider accepted the charge. It needs an honest intermediate state.

Decision

The workflow uses a short seat hold before payment.

1. Customer selects A-14.
2. Seat owner creates hold-82 with an expiry time.
3. Payment service attempts pay:hold-82.
4. Ticket issuer creates ticket-501 only after hold and payment evidence agree.
5. Delivery services send email and wallet updates from ticket_issued events.

The browser may retry, but retries carry the same purchase_intent and payment_operation.

retry with pay:hold-82
  -> payment service returns existing outcome if known
  -> does not create a second provider charge

The user-visible states are explicit:

seat_available
seat_held
payment_authorizing
purchase_pending
ticket_issued
hold_expired
needs_reconciliation
purchase_unavailable

purchase_pending is important. It means:

the system has a durable purchase intent
but does not yet have enough evidence to issue or reject the ticket

It is not a failure. It is a promise to repair from evidence.

Normal Path

In the normal path, the facts line up.

10:00:00 seat owner:
  hold-82 created for event-9/A-14/customer-7
  expires_at=10:05:00

10:00:02 payment:
  pay:hold-82 authorized

10:00:03 ticket issuer:
  hold-82 + pay:hold-82 -> ticket-501 issued

10:00:04 delivery:
  ticket_issued v1 delivered to email and wallet workers

The system can show "ticket issued" only after the ticket issuer records the final ticket. Email delivery is not the authority. If the email queue is slow, the ticket still exists and delivery can be retried or repaired.

Ambiguous Timeout Path

Now the payment provider accepts pay:hold-82, but the payment service times out before receiving the response.

Naive decision:

tell the browser payment failed
let the browser create a new payment operation

That can duplicate charges.

Better decision:

record purchase_pending
reuse pay:hold-82 on retry
query provider or wait for callback
issue ticket only after one durable payment outcome exists

Intermediate state:

seat: held for hold-82
payment: unknown
ticket: not issued
user state: purchase_pending
repair: provider lookup for pay:hold-82

Output:

The customer sees an honest pending state. The system does not sell the seat to someone else while the hold is still valid, and it does not create another charge for the same intent.

Naive failure contrast:

If unknown had been treated as failed, the customer might retry with a new payment operation while the provider later confirms the first one. If unknown had been treated as success, the system might issue a ticket without durable payment evidence.

Partition Path

Suppose the edge API can reach payment but cannot reach the seat owner.

The edge API must not independently sell seat A-14. That would create a second authority for the same final fact.

Safe behavior:

if seat owner unreachable:
  do not confirm seat sale
  preserve search or cart state if possible
  return purchase_unavailable or retry-after

This is a CAP-style trade-off in a concrete form. During a partition, the workflow chooses not to remain fully available for final seat purchase because the seat must not fork. The system may keep browsing available, but final seat ownership requires the owner path.

Overload And Degraded Mode

During the first sale minute, requests spike.

Signals:

hold request arrival rate > completion rate
oldest hold queue age rising
payment provider p99 latency rising
browser retry rate rising
wallet delivery queue age rising

The system enters a sale-degraded mode.

preserve:
  existing holds
  payment status lookup
  ticket status lookup
  repair workers

shed or degrade:
  seat map refresh frequency
  recommendation widgets
  nonessential wallet enrichment
  repeated hold attempts from the same account

reject:
  new hold attempts when queue age exceeds the useful hold budget

The smaller promise is:

We will protect existing purchase intents and show truthful status.
New purchases may be delayed or refused until capacity is safe.

This is backpressure tied to product truth. It refuses work before the system creates stale holds and ambiguous payments.

Check: Why should wallet delivery be degraded before seat ownership or payment repair?

Think first, then reveal.

Answer: Wallet delivery is important, but it is not the authority for seat ownership or payment. The system must preserve the facts that decide whether the customer owns the seat before spending scarce capacity on optional delivery enrichment.

Contracts And Versioned Messages

The ticket issuer publishes:

ticket_issued
  ticket_id
  event_id
  seat_id
  customer_id
  issued_at

Later, the team adds:

entry_gate

The safe rollout prepares readers first. Old wallet workers must ignore the optional field, or the producer must wait until they are upgraded. New wallet workers must still read old ticket_issued messages without entry_gate, especially during replay.

The semantic meaning must also stay stable. If seat_id used to mean the physical assigned seat, it must not silently become a pricing-zone id. That kind of change can pass a schema check while breaking venue entry.

Retirement evidence:

no old wallet workers reading live queues
replay path tested for old and new ticket_issued messages
dead-letter volume normal
fallback use for missing entry_gate near zero
owner approves removal of old reader behavior

Evidence To Retain

Every boundary keeps joinable evidence.

request_id:          one browser/API attempt
trace_id:            one observed execution path
purchase_intent:     one customer intent
reservation_id:      one seat hold
payment_operation:   one external payment side effect
ticket_id:           one issued ticket
message_id:          one queue delivery attempt
repair_id:           one reconciliation action

Logs show local decisions. Traces show where the path waited. Metrics show pressure: queue age, retry rate, provider latency, dead-letter rate, and repair backlog. Durable records show official state: seat hold, payment outcome, ticket issue, delivery status, and reconciliation result.

Support should be able to start from customer-7 or seat A-14 and answer:

Was the seat held?
Was payment authorized?
Was a ticket issued?
Was delivery delayed?
Is repair still pending?
What evidence supports the final answer?

Consequences And Open Risks

This design protects double-sale and duplicate-charge promises. It gives customers an honest pending state when evidence is incomplete. It gives operators a repair path and enough evidence to explain one case.

The costs are real:

Open risks remain:

These risks do not make the design bad. They make it reviewable. A reviewer can now ask for tests, metrics, owners, and rehearsal evidence.

Review The ADR By Pressure

After writing the ADR, review it by pressure rather than by component.

Promise Pressure

Ask:

What user-visible promise would be embarrassing or harmful to break?

For tickets, the promise is not "the API returns 200." It is:

one seat is sold once
one purchase intent creates at most one charge
the customer sees a truthful status while evidence is incomplete

If an answer starts with a database, cache, queue, or cloud region before naming this promise, it is probably starting too low.

Ownership Pressure

Ask:

Who owns the final fact?

The seat owner decides whether A-14 is held or sold. The payment service records the outcome of pay:hold-82. The ticket issuer decides whether a QR ticket exists.

Those facts may be copied elsewhere, but copies are not authorities. A wallet app can display a ticket. It should not invent one. An email worker can notify the customer. It should not decide that payment succeeded.

This is where replication and consistency become concrete. Some reads may be stale. Final state transitions should not silently fork.

Retry Pressure

Ask:

Which retry might duplicate a side effect?

A browser retry should reuse purchase_intent. A payment retry should reuse payment_operation. A ticket-delivery retry should reuse ticket_id and message identity. Each receiver needs enough durable memory to say:

I have already seen this logical operation.
Here is the outcome I recorded.

If the design says "retry" but cannot name the stable operation identity, the retry is not yet safe.

Load Pressure

Ask:

What happens when demand stays above completion rate?

The seat hold queue cannot grow forever. A hold has a useful lifetime. A hold request that waits ten minutes during a five-minute sale may no longer be useful or fair. The design needs queue age, admission rules, and a smaller promise under overload.

This is the same backpressure idea from earlier lessons, but now it is tied to a capstone artifact. The ADR should say which work is preserved, which work is shed, and which user-facing response is honest.

Contract Pressure

Ask:

What old data or old code can still appear after the deploy?

The ticket_issued event may sit in queues, dead-letter stores, archives, or replay tools. A safe design names the compatibility window. It also distinguishes shape from meaning. Adding entry_gate is a shape change. Reusing seat_id to mean "pricing section" would be a semantic change and should not be hidden behind the old field name.

Recovery Pressure

Ask:

What evidence proves the system can return to normal mode?

It is not enough for the payment provider to look healthy. The system also needs pending purchases drained, repair actions completed or bounded, retry rate normal, dead-letter messages understood, and sample traces that join seat, payment, ticket, and delivery facts.

Recovery is a state transition too. It deserves evidence.

Practice: Tighten A Weak ADR

Here is a weak capstone answer:

We will put ticket purchases in a queue, retry payment failures,
replicate the database to two regions, and add dashboards.

Rewrite it using the review frame:

promise:
ownership:
uncertainty:
mechanisms:
evidence:
trade-off:

Model answer:

promise:
  one seat is sold once, and one purchase intent creates at most one charge

ownership:
  seat owner decides hold/sold state; payment service owns provider outcome;
  ticket issuer owns QR ticket creation

uncertainty:
  payment timeout may mean accepted, rejected, or unknown;
  replicated reads may lag; delivery queues may be delayed

mechanisms:
  stable purchase_intent and payment_operation, owner-backed seat transition,
  purchase_pending state, bounded queues, degraded sale mode,
  compatible ticket_issued rollout

evidence:
  request id, trace id, reservation id, payment operation, ticket id,
  outbox events, queue age, repair status

trade-off:
  final purchase may be delayed or unavailable under partition or overload,
  but the system avoids double sale, duplicate charge, and false confirmation

This is not much longer than the weak answer. It is much easier to review.

Capstone Deliverable

Write your own ADR for the ticket workflow or for a different workflow: password reset, file upload, seat booking, message send, account deletion, or live quiz grading.

Use:

# ADR: <Workflow> Under Partial Failure

## Context
## Decision
## Consequences
## Evidence To Retain
## Open Risks

Your ADR is ready when a reviewer can answer:

Final Self-Review

Before you consider the capstone complete, read your ADR once as a customer and once as an operator.

As a customer, ask:

Will I know whether I own the seat?
Will I avoid being charged twice for one intent?
Will the system tell me the truth if it is uncertain?

As an operator, ask:

Can I find the official seat state?
Can I find the payment operation outcome?
Can I see whether a queue, retry loop, or old message version is blocking progress?
Can I decide when degraded mode may safely end?

If either reader needs guesswork, improve the ADR. A capstone answer does not need to be long, but it must make the hard parts visible enough for another person to challenge them.

What This ADR Does Not Do

A good ADR also says what it is not deciding.

This ticket ADR does not choose a specific database product, cloud vendor, message broker, or payment provider. Those choices matter, but they come after the workflow promise is clear. It also does not prove that every customer will get a ticket during a spike. In fact, it deliberately refuses some purchases when accepting them would create stale holds or ambiguous payment work.

It does not remove the need for load testing, schema compatibility tests, payment-provider contract review, or incident rehearsals. The ADR makes those checks easier to name. It turns hidden assumptions into reviewable obligations.

That boundary is healthy. A foundation-level design should be precise about behavior before it becomes specific about vendors.

Rubric

Strong capstone answers:

Weak answers:

Resources

Key Takeaways

PREVIOUS Foundations Review Check
NEXT End of track

← Back to Distributed Systems Foundations

← Back to Distributed Systems

← Back to Learning Hub