Capstone: Design an Event-Driven Service Platform

By the end of this lesson, you will be able to...

• Design an event-driven service boundary by naming facts, owners, consumers, topology, delivery behavior, and operational signals.

• Defend retry, idempotency, schema, replay, DLQ, and backpressure decisions for one realistic platform scenario.

• Explain which concerns belong in this architecture track and which should move to deeper tracks on messaging internals, CDC, stream processing, sagas, or workflows.

Idea in one sentence: A good event-driven platform design does not say "we publish events"; it says which facts move, who owns them, what can repeat, what can replay, and which signal proves the user promise is still healthy.

Core Insight

MarketplacePlatform lets buyers place orders from many sellers. The current checkout service calls every downstream system directly:

checkout -> inventory
checkout -> payment
checkout -> seller notification
checkout -> email
checkout -> analytics

This works while traffic is small.

Then a sale starts.

Inventory is slow. Payment has a temporary timeout. Email accepts requests but later throttles. Analytics falls behind. The checkout team says:

Let's make it event-driven.

That sentence is only a starting point. It does not yet describe an architecture.

An event-driven design becomes real when the team can answer smaller questions:

• What fact happened?
• Which service is allowed to say that fact is true?
• Which consumers need the fact?
• Should consumers compete for work or each receive a copy?
• What happens when delivery repeats?
• Which actions are safe to replay?
• How will old consumers survive new fields?
• Which lag signal maps to user harm?
• Which deeper problem is outside this track?

This capstone asks you to design the event boundary for the marketplace platform. The goal is not to use every event pattern. The goal is to make a small architecture defensible.

The Design Brief

The product team wants three user promises:

1. A buyer should see an accepted order quickly after payment authorization succeeds.
2. A seller should receive one actionable order notification.
3. Reports should eventually include every accepted order, even if analytics is delayed.

The platform has five services:

The naive event proposal is:

checkout-service publishes OrderSubmitted
all other services subscribe

This proposal is too vague.

OrderSubmitted might mean the buyer clicked "Buy". It might mean checkout received the request. It might mean validation passed. It might mean payment was authorized. It might mean the order is ready for fulfillment.

Those meanings have different owners and different risks.

The capstone design should start from one durable fact:

OrderAccepted

Plain meaning:

The platform has accepted an order after checkout validation and payment authorization succeeded.

The owner is checkout-service, because checkout records the accepted order and can prove the fact before publishing it.

A Worked Design Path

Here is one defensible path through the design.

The output is not just a diagram. It is a set of decisions.

One possible event contract:

OrderAccepted(
  event_id,
  order_id,
  buyer_id,
  seller_id,
  accepted_at,
  payment_authorization_id,
  total_cents,
  currency,
  schema_version
)

One possible topology:

checkout-service
  -> order-events topic/log
       -> seller-service consumer
       -> email-service consumer
       -> analytics-service consumer

One possible delivery policy:

At-least-once delivery is expected.
Consumers must make side effects idempotent.
DLQ is allowed only after bounded retries and visible alerting.
Replay is allowed by default for analytics projection.
Replay is not allowed by default for external emails or seller task creation.

This is the shape of a capstone answer. Each choice names the fact, the owner, the consumer behavior, and the cost.

Check: Why does the design use fanout rather than one shared work queue for OrderAccepted?

Think first, then reveal.

Answer: Seller tasks, emails, and analytics all need to observe the same accepted-order fact. They are not interchangeable workers doing one copy of the same job. Fanout gives each subscriber its own observation path. A shared competing queue would let one service consume the message and hide it from the others.

Defending the Design

In a capstone review, a design is not finished when the boxes are connected. It is finished when the team can defend why each box exists and what happens when the happy path breaks.

Use a short defense like this:

We publish OrderAccepted because checkout owns the accepted-order fact.
We use fanout because seller work, email, and analytics each need the same fact.
We expect duplicate delivery, so each consumer has an idempotency boundary.
We replay analytics by default because it is derived state.
We do not replay email or seller task creation by default because those create side effects.
We watch oldest seller task age, email retry rate, analytics lag, and DLQ count because those signals map to user promises.

Notice what this defense does.

It does not say the design is perfect. It says the design has named failure boundaries. That is a much stronger claim.

If a reviewer asks, "What if the broker stores the event but email is down?", the answer is not "the broker handles it." The answer is:

email-service lag and retry rate rise.
checkout does not block on email.
the buyer order is still accepted.
the email consumer retries with idempotency.
after bounded retries, failing events move to DLQ with an alert.

If a reviewer asks, "What if analytics missed yesterday's events?", the answer is:

analytics-service can replay OrderAccepted from the log into its projection.
the replay targets derived state only.
seller-service and email-service are not included in that replay unless a separate recovery plan approves it.

This is the difference between using events and designing with events.

Using events means messages move.

Designing with events means the team can explain timing, ownership, repetition, recovery, and evidence.

The Architecture Review Rubric

Use this rubric to review your design.

1. Fact and ownership

A strong design says:

OrderAccepted is a durable business fact.
checkout-service owns it.
checkout publishes it only after recording the accepted order.

A weak design says:

Something happened in checkout, so publish an event.

The difference matters because consumers treat facts as evidence. If the event is published before the fact is true, downstream services can create real side effects from a false or premature signal.

2. Topology and consumer independence

A strong design explains why subscribers each need a copy:

seller-service creates seller work.
email-service sends notification.
analytics-service updates projection.

Those are different jobs. They should not compete for one message.

The trade-off is cost and operational surface. Each subscriber now has its own lag, retries, and idempotency records. Decoupling the checkout request did not remove responsibility. It moved responsibility into event contracts and consumer operations.

3. Delivery and side effects

A strong design assumes duplicate observation.

The broker may redeliver because a consumer crashes before acknowledgment. The consumer may process the event but fail before saving its progress. The network may timeout after a side effect already happened.

So each consumer needs a rule:

This does not make the whole system exactly-once. It makes repeated delivery safe enough at the application boundary.

4. Contract evolution

A strong design defines what can change without breaking consumers.

Safe changes:

• Add optional fields.
• Preserve old field meaning.
• Publish a new event type when meaning changes.
• Run old and new consumers during migration.

Unsafe changes:

• Rename a field used by active consumers.
• Change total_cents from order total to discounted subtotal without a new name.
• Remove seller_id before every consumer has migrated.
• Treat schema parsing as proof that business meaning is compatible.

The hard part is semantic compatibility. A consumer can parse a message and still misunderstand it.

5. Replay, DLQ, and recovery

A strong design separates replay-safe work from side-effect work.

Analytics projection is a natural replay target because it derives state from event history. If the projection is wrong, rebuild it from the log.

Email and seller task creation are different. They create user-visible or operational side effects. Replay may be possible, but only with idempotency, target filters, dry runs, and reconciliation.

DLQ policy should also be explicit:

After 5 retries, move malformed or repeatedly failing events to DLQ.
Alert with event type, consumer, oldest failed age, and sample event id.
Do not silently drain DLQ into production consumers.
Replay from DLQ through a reviewed recovery path.

The trade-off is delay versus harm. Retrying forever hides poison events and increases lag. Sending to DLQ too early can pause recoverable work. The design should state the boundary.

6. Operational signals

A strong design maps signals to promises:

Do not alert only on "broker backlog is high." That signal is too broad. A useful signal tells the team which promise is degrading and which consumer owns the response.

Also separate platform symptoms from product symptoms.

Broker backlog is a platform symptom.
Delayed seller fulfillment is a product symptom.
Email retry rate is a platform symptom.
Missing buyer confirmation is a product symptom.

The platform symptom tells you where to investigate. The product symptom tells you why anyone should care.

That distinction improves incident response. If analytics lag is high, the urgent question is not "is the broker large?" The urgent question is "are business reports stale enough to cause bad decisions?" If seller task age is high, the urgent question is "which sellers cannot start fulfillment yet?" The event platform should make those questions easy to answer.

Good event-driven operations does not stop at moving messages. It connects message movement to the promise the system made.

Check: A replay request says, "Replay all OrderAccepted events from the last 24 hours through every consumer." What should you challenge first?

Think first, then reveal.

Answer: Challenge the phrase "every consumer." Analytics may be replay-safe, but email and seller task creation have external or operational side effects. The design should name which consumers can replay, what idempotency protects them, which time window is safe, and how the team will verify the result.

Boundary to Follow-On Tracks

This capstone should expose deeper concerns without solving all of them here.

This boundary keeps the capstone honest. A good answer can say:

This design chooses an application event boundary.
It does not yet design broker replication or stream joins.
Those are follow-on design reviews.

That is not weakness. It is scope control.

Practice

Design the smallest event-driven architecture for this change:

After an order is accepted, the platform must:
1. create one seller fulfillment task,
2. send one buyer confirmation email,
3. update analytics eventually,
4. tolerate duplicate delivery,
5. recover analytics after a projection bug,
6. show operations which user promise is delayed.

Write a short design note with these headings:

Fact and owner
Topology
Consumer idempotency
Replay and DLQ policy
Schema evolution rule
Operational signals
Deferred deeper tracks

Model answer:

Fact and owner:
checkout-service publishes OrderAccepted after it records the accepted order and payment authorization id.

Topology:
OrderAccepted goes to an order-events topic/log. seller-service, email-service, and analytics-service each have independent subscriptions.

Consumer idempotency:
seller-service creates one task per order_id. email-service records one send per template, order_id, and recipient. analytics-service upserts projection rows by order_id.

Replay and DLQ policy:
analytics can replay from the event log after projection bugs. seller and email do not replay by default without a reviewed recovery plan. Malformed or repeatedly failing events move to DLQ after bounded retries and alerting.

Schema evolution rule:
add optional fields first, preserve old meaning, and create a new event type when meaning changes.

Operational signals:
alert on oldest seller task age, email retry rate, analytics lag, DLQ count by event type, and duplicate idempotency hits.

Deferred deeper tracks:
broker replication and consumer group internals move to messaging internals. Stream joins and exactly-once projection depth move to streaming infrastructure. Compensation across payment and fulfillment moves to sagas or workflows.

Use the rubric to grade your own answer:

Resources

• [BOOK] Martin Kleppmann - Designing Data-Intensive Applications
  • Link: https://dataintensive.net/
  • Focus: Use for logs, derived data, messaging, consistency, and the trade-offs behind replay and event-driven data flow.
• [ARTICLE] Martin Fowler - What do you mean by Event-Driven?
  • Link: https://martinfowler.com/articles/201701-event-driven.html
  • Focus: Use for distinguishing event notification, event-carried state transfer, event sourcing, and CQRS in architecture discussions.
• [ARTICLE] Chris Richardson - Transactional Outbox
  • Link: https://microservices.io/patterns/data/transactional-outbox.html
  • Focus: Use for the boundary between local database commits, publication responsibility, and reliable event emission.

Key Takeaways

• The capstone design starts with a durable fact and a named owner, not with a broker product or a generic event name.
• Fanout, retry, idempotency, replay, schema evolution, DLQ behavior, and operational signals are part of the application architecture contract.
• Duplicate delivery is normal enough that every side-effecting consumer needs an idempotency boundary.
• Replay is not automatically safe for every consumer; derived projections and external side effects need different recovery rules.
• A strong event-driven design states what is intentionally deferred to messaging internals, CDC, stream processing, sagas, or workflow orchestration.