Producers, Consumers, Brokers, and Event Logs

By the end of this lesson, you will be able to...

• Trace how a produced domain fact moves through a broker or event log to several consumers.

• Explain which state is owned by the producer, broker, and consumer at each step.

• Predict what can still fail after the broker has accepted an event.

Idea in one sentence: A broker or event log does not make an event-driven system magic; it gives producers and consumers a shared place to hand off facts while each side keeps its own responsibilities.

Core Insight

Suppose the order service has just stored order O-812.

From the previous lesson, we know the useful message is a fact:

OrderPlaced

Payment wants to react. Email wants to react. Analytics wants to react. The naive picture is simple:

order service publishes OrderPlaced
everyone receives it
done

That picture hides the mechanism.

Who has the event before the broker accepts it? Who has it after? What if payment is offline? What if email handles the event twice? What if analytics starts later and wants old events?

The core idea in this lesson is that an event flow has moving parts with different ownership:

producer -> broker or event log -> consumer

The producer owns the fact it is allowed to publish. The broker or log owns durable handoff and delivery position. Each consumer owns its reaction, its progress, and its side effects.

If you blur those responsibilities, event-driven design becomes hard to debug. A team may say "the event was published" when the producer only tried to send it. Another team may say "payment received the event" when the broker only made it available. Those are different states.

The Situation

Our checkout system has four services:

Order service
Payment service
Email service
Analytics service

The order service owns the order record. When it commits order O-812, it can publish this integration event:

{
  "type": "OrderPlaced",
  "event_id": "evt-1001",
  "order_id": "O-812",
  "customer_id": "C-44",
  "total": "42.00",
  "currency": "EUR",
  "occurred_at": "2026-07-08T10:15:00Z"
}

Payment should attempt capture. Email should prepare a receipt. Analytics should count the order.

The first design a team often imagines is direct fanout:

Order service -> Payment
Order service -> Email
Order service -> Analytics

That can work in small systems, but it couples order creation to every downstream service. If email is down, does order placement fail? If analytics is slow, does checkout wait? If payment needs retries, does the order service own those retries?

An event broker or event log adds a handoff point:

Order service -> broker/log -> Payment
                         -> Email
                         -> Analytics

This does not remove coupling. It changes where the coupling is visible.

The Moving Parts

Plain meaning:

A producer is the part that writes a message into the event system.

In this scenario:

The order service is the producer for OrderPlaced.

Technical name:

Producer means "the component that sends or appends the event." It does not mean "the owner of every reaction." The order service owns the fact that an order was placed. It does not own payment capture, receipt delivery, or analytics counting.

Plain meaning:

A consumer is the part that reads the message and decides what to do with it.

In this scenario:

Payment, email, and analytics are consumers of OrderPlaced.

Technical name:

Consumer means "the component that observes the event and advances its own state." A consumer may update a database, send another command, write a projection, or ignore the event if it is not relevant.

Plain meaning:

A broker is the handoff system between producers and consumers.

In this scenario:

The broker accepts OrderPlaced, stores enough information to deliver it, and makes it available to interested consumers.

Technical name:

Broker is the middleware that receives messages and coordinates delivery, routing, retention, acknowledgments, or offsets. Different products do this differently. This lesson stays at the application-architecture level.

Plain meaning:

An event log is an ordered record of events that consumers can read from a position.

In this scenario:

OrderPlaced is appended at position 1057. Payment may read up to 1057, email may be behind at 1053, and analytics may replay from 1000.

Technical name:

An event log is an append-oriented stream. Consumers track progress through offsets or similar positions. A log makes replay and independent consumer progress easier to reason about, but it also makes retention, ordering scope, and duplicate handling explicit.

The Mechanism Step by Step

Here is the small mechanism.

Starting state:

Order database:
  O-812 does not exist yet

Broker/log:
  topic or stream: order-events
  latest position: 1056

Consumers:
  Payment offset: 1056
  Email offset: 1056
  Analytics offset: 1050

The customer clicks "Place order."

Step 1: the order service commits its own state.

Order database:
  O-812 = PLACED

At this moment, the order service owns a new fact. No consumer has seen it yet.

Step 2: the order service produces the event.

produce OrderPlaced(evt-1001, O-812)

The producer is asking the broker or log to accept a durable handoff. Until the broker confirms, the event may still be only in the producer's memory, request buffer, or outbox table.

Step 3: the broker or log accepts it.

order-events position 1057:
  OrderPlaced(evt-1001, O-812)

Now the handoff state has changed. The event is no longer just something the order service tried to send. It is available in the event system.

Step 4: each consumer reads from its own position.

Payment reads from 1057
Email reads from 1057
Analytics reads from 1051, then eventually reaches 1057

The consumers do not all move together. Payment may be current. Email may be paused. Analytics may be replaying old events.

Step 5: each consumer performs its own reaction.

Payment:
  sees OrderPlaced
  creates payment attempt P-55
  records "handled evt-1001"
  advances offset to 1057

Email:
  sees OrderPlaced
  enqueues receipt email
  records "handled evt-1001"
  advances offset to 1057

Analytics:
  eventually sees OrderPlaced
  increments order count
  advances offset to 1057

The broker can help deliver the event. It cannot decide whether payment capture is correct. It cannot make email idempotent. It cannot guarantee that analytics chose the right metric.

So far, we have a better picture than "publish and done." An event moves through several states:

fact committed -> event produced -> broker accepted -> consumer read -> consumer side effect -> consumer progress recorded

Each arrow can fail in a different way.

A Worked Trace

Now trace one event with an uneven system. Payment is healthy. Email is down for five minutes. Analytics is behind because it is replaying.

Input:

O-812 is stored as PLACED

Transition:

The order service produces OrderPlaced(evt-1001).

Intermediate states:

Output or decision:

The order service did not wait for every consumer. The broker or log held the event long enough for independent consumers to catch up. Each consumer recorded its own progress.

Naive failure contrast:

In the direct-call design, email downtime could force the order service to choose between blocking checkout, dropping the receipt request, or owning email retry logic. With broker mediation, email can be unavailable while the order fact remains available for later handling.

That is the benefit. But notice the boundary: the broker did not prove the receipt was sent. It only made the event available and let email resume from a known position.

Check: At T2, can the order service honestly say "payment captured the money"?

Think first, then reveal.

Answer: No. At T2, the order service can say it committed the order and the broker accepted OrderPlaced. Payment has read the event and created an attempt, but the payment service owns whether capture succeeds. A later PaymentCaptured fact would need to come from the payment service.

Where It Breaks

The mechanism helps, but it has sharp edges.

Producer failure before broker acceptance

The order service may commit O-812, then crash before the broker accepts OrderPlaced.

Now the database says the order exists, but consumers never see the event.

This is not solved just by adding a broker. Later in the track, the outbox pattern will make this gap visible and repairable. For now, name the state precisely:

fact committed
event not yet durably handed off

Duplicate consumer handling

The broker may deliver the same event more than once, or a consumer may crash after doing work but before recording its progress.

Email might enqueue a receipt, crash, restart, and read evt-1001 again.

The better model is:

broker delivery can be repeated
consumer side effects must be safe enough for repeats

That usually means the consumer records an event id, uses an idempotency key, or designs the side effect so repetition is harmless.

Consumer lag

Analytics may fall behind:

latest broker position: 1057
analytics offset: 1051
lag: 6 events

Lag is not automatically a bug. It may be expected during replay or a traffic spike. It becomes a problem when the product promise depends on fresh analytics, retention is about to expire, or downstream storage cannot catch up.

Retention and replay limits

An event log is not infinite unless you pay for it and operate it that way.

If analytics starts from offset 1000 but the log only retains events from 1030, replay cannot reconstruct the missing slice from the broker alone.

This is a boundary of event logs: they make replay possible within the retained history. They do not guarantee that old history exists forever.

Cost, Limits, and Signals

Broker or log mediation improves independent progress. Payment, email, and analytics can move at different speeds.

It costs operational complexity. You now have broker availability, retention policy, consumer offsets, lag, duplicate delivery, schema compatibility, and replay behavior to review.

It can still fail when the producer commits local state but never hands off the event, when a consumer performs a side effect twice, or when a slow consumer falls behind retention.

It does not solve ownership. The order service still cannot publish PaymentCaptured just because it wants payment to happen. The broker carries messages; it does not change which service owns a fact.

Signals to watch:

• producer publish failures or outbox backlog
• broker append or publish latency
• consumer lag by stream, topic, partition, or subscription
• repeated delivery or retry counts
• dead-letter counts for events that cannot be handled
• consumer side-effect errors after successful reads

The trade-off is clear: the event system reduces direct timing coupling between services, but it introduces a durable handoff surface that must be designed, monitored, and tested.

Common Confusions

Confusion: "Published" means "processed"

Why it is tempting:

The producer receives a successful publish response and the team wants a simple status word.

Better model:

Published means the event system accepted the event. Processed is a consumer-owned fact. Different consumers may process the same event at different times.

Confusion: The broker owns business correctness

Why it is tempting:

The broker is central in the diagram, so it feels like the broker controls the workflow.

Better model:

The broker owns handoff mechanics. Business correctness still lives in producers and consumers: what fact was emitted, who is allowed to emit it, which side effects are safe, and how duplicates are handled.

Confusion: A log means every consumer sees the same system

Why it is tempting:

The log gives one ordered record, so it feels like everyone has the same view.

Better model:

The log may contain one ordered record, but each consumer has its own position. Payment may know about 1057 while analytics is still at 1052. Shared history does not mean shared current knowledge.

Trace It Yourself

Check: Email reads OrderPlaced(evt-1001), sends the receipt, then crashes before recording offset 1057. What should you expect after restart?

Think first, then reveal.

Answer: Email may read evt-1001 again. The broker or log may only know that email did not record progress. The email consumer needs a safe side-effect strategy, such as storing evt-1001 as already handled or using a receipt idempotency key.

Now try a nearby case.

A fulfillment service consumes OrderPlaced and creates a warehouse pick request.

Trace this event:

OrderPlaced(evt-2040, order_id=O-991)

Starting state:

broker latest position: 2200
fulfillment offset: 2199
warehouse API: sometimes times out after creating the pick request

Write a five-step trace that includes:

• where the event is durably stored
• when fulfillment reads it
• what happens if the warehouse API times out
• what fulfillment records before advancing its offset
• what signal would show that fulfillment is falling behind

A good answer should mention that the broker can retain evt-2040, but fulfillment owns the warehouse side effect. If the warehouse API times out after creating the pick request, fulfillment needs an idempotency key or a way to check whether the pick request already exists before retrying. The lag signal is the distance between the broker's latest position and fulfillment's recorded position.

Resources

• [BOOK] Designing Data-Intensive Applications - Martin Kleppmann
  • Link: https://dataintensive.net/
  • Focus: Read the messaging, logs, and derived data discussions for the difference between handoff, history, and consumer progress.
• [ARTICLE] What do you mean by Event-Driven? - Martin Fowler
  • Link: https://martinfowler.com/articles/201701-event-driven.html
  • Focus: Use the distinctions between event notification, event-carried state transfer, event sourcing, and CQRS to keep message meaning separate from transport.
• [DOC] Apache Kafka Documentation: Introduction
  • Link: https://kafka.apache.org/documentation/#introduction
  • Focus: Look at topics, producers, consumers, and offsets as one concrete implementation of the producer-log-consumer model.

Key Takeaways

• A producer owns the fact it is allowed to publish; it does not own every downstream reaction.
• A broker or event log owns durable handoff and delivery position, not business success.
• Consumers move independently, so "event accepted" and "consumer processed" are different states.
• Event logs make replay and lag visible within retained history; they do not make history infinite.
• The main trade-off is less direct timing coupling in exchange for more handoff, offset, duplicate, and operational responsibility.