Queues, Topics, Pub/Sub, and Log Topologies

By the end of this lesson, you will be able to...

• Compare queue, topic, pub/sub, and log topologies from application needs.

• Choose a topology by naming fanout, retention, replay, ordering, and ownership constraints.

• Review the operational signal that shows when the chosen topology is under pressure.

Idea in one sentence: A messaging topology is a design choice about who should receive a fact, how long the fact should remain useful, and whether consumers need independent history or shared work distribution.

Core Insight

Imagine the order service now publishes a truthful OrderPlaced event.

The previous lesson gave us the boundary rule:

The order service owns OrderPlaced.
Consumers own their reactions.

Now a new question appears:

How should that event move?

The first naive answer is, "Put it on a queue." That answer is too small. A queue is one shape, but event-driven architecture often needs several different shapes.

For example:

• Payment needs to process each order once as work.
• Email needs to know about every placed order, even if payment also sees it.
• Analytics may need to read all order events from last night again.
• A search projection may need an ordered history for each product or order.

Those are not the same delivery problem.

The core design rule is:

Choose the topology from the consumer relationship, not from the broker product name.

A topology is the shape of movement between producers, brokers or logs, and consumers. It decides whether consumers compete for work, each receive their own copy, subscribe through a filtered channel, or read durable history at their own pace.

This lesson stays at the application architecture level. Broker internals, partition implementation, replication, and vendor-specific features belong in later messaging tracks. Here we need the review model: what does this shape promise, what does it cost, and what signal tells us it is unhealthy?

The Small Situation

Our commerce system has one producer:

Order service publishes OrderPlaced

Four consumers want to react:

If all four share one simple queue, something odd happens.

One consumer receives a message and the others do not. That is great for distributing one unit of work among many workers. It is wrong when several independent systems all need to learn the same fact.

If all four receive every event but the broker discards each event immediately after delivery, analytics cannot replay yesterday's events when a reporting bug is fixed.

If every event is kept forever in one ordered log, payment might be forced into a history model it does not need for day-to-day work.

The design is not "which tool is best?" The design is "which movement shape matches this consumer relationship?"

Plain Meaning, Then Precise Meaning

Plain meaning:

A queue is a line of work. One message should be handled by one worker from a group.

In this scenario:

Payment might use a queue of CapturePayment commands or payment tasks so that one payment worker handles each order.

Technical name:

A work queue distributes messages among competing consumers. It is useful when workers are interchangeable and the goal is shared work, not independent observation.

Plain meaning:

A topic is a named channel. Many subscribers can receive messages published to that channel.

In this scenario:

orders.placed can be a topic where email, analytics, and fraud each receive OrderPlaced.

Technical name:

A topic is a publish-subscribe routing abstraction. It lets the producer publish once while multiple subscriptions receive independently.

Plain meaning:

Pub/sub means the producer does not send directly to each consumer. Consumers subscribe to what they need.

In this scenario:

The order service does not call email, analytics, and fraud. It publishes OrderPlaced, and each consumer subscribes.

Technical name:

Publish-subscribe is an interaction style where producers publish messages to an intermediary and subscribers receive matching messages according to subscriptions or routing rules.

Plain meaning:

A log is a durable sequence of records. Consumers can remember their own position and read old records again if retention allows it.

In this scenario:

Analytics may read every OrderPlaced event from offset 4,000 to offset 9,000 again after a bug fix.

Technical name:

An append-only event log stores ordered records for some retention period. Consumers track offsets or positions, which makes independent replay and backfill possible.

The Naive Design

The team starts with this design:

Order service -> one queue -> payment worker
                         -> email worker
                         -> analytics worker
                         -> fraud worker

It looks tidy because there is one place to put events.

But a queue has a specific meaning. If these workers are competing consumers, each OrderPlaced message is taken by one worker. Payment might receive order O-812; email might receive O-813; analytics might receive O-814.

That is shared work distribution, not broadcast.

The naive design fails because it confuses two questions:

Question 1: Should one of many workers handle this unit of work?
Question 2: Should many independent systems all observe this fact?

Those questions need different shapes.

Design Alternatives

Here are four common topologies at the application level.

These shapes can be combined.

A topic may feed a subscription per consumer. Each subscription may behave like a queue for that consumer's worker pool. A log may expose the same retained event history to multiple consumer groups. The product vocabulary differs across brokers, but the design questions are stable:

• Is this a fact that many systems must observe?
• Is this a command or task that one worker should perform?
• Does each consumer need its own position?
• How long must the event remain available?
• Can the consumer rebuild state by replaying events?
• What ordering scope matters?

Do not choose the strongest shape everywhere. Durable logs and replay are powerful, but they add retention, storage, offset, schema, and operational duties. A simple work queue may be better for short-lived tasks that do not need independent history.

A Worked Topology Choice

Now design movement for OrderPlaced.

Input:

{
  "type": "OrderPlaced",
  "event_id": "evt-1001",
  "order_id": "O-812",
  "customer_id": "C-44",
  "occurred_at": "2026-07-08T10:15:00Z"
}

Transition:

The order service publishes the owned fact once.

Intermediate state:

Different consumers need different relationships to that fact.

Output or decision:

One reasonable architecture is:

Order service
  -> publish OrderPlaced to orders.placed

orders.placed
  -> email subscription
       -> email worker pool
  -> analytics consumer group
       -> retained event history and offsets
  -> fraud subscription
       -> fraud worker pool
  -> payment policy consumer
       -> CapturePayment command queue

Naive failure contrast:

If the team used one competing queue for all consumers, only one consumer type would see each order. If it used only direct calls, the order service would again know every downstream reaction. If it used a durable log for every tiny command, operational cost might grow without a matching need for replay.

So far:

Queue = one of these workers should do this work.
Topic/pub/sub = each interested subscriber should see this fact.
Log = each consumer may need durable history and its own read position.

Check: The email service runs ten identical workers. Should OrderPlaced be delivered to all ten workers?

Think first, then reveal.

Answer: Usually no. Email as a consumer should receive every relevant OrderPlaced event, but the ten email workers should compete inside the email subscription or queue so one worker sends one email for one order.

Trade-offs and Limits

Topology choice improves clarity because it separates broadcast, shared work, filtering, and replay.

It costs operational ownership. Each shape creates a different backlog, retry, retention, and monitoring model.

The trade-off is direct: a more capable topology can make consumers more independent, but it also creates more state for the team to operate and explain.

A work queue keeps worker pools simple, but it does not give independent consumers their own copy of each event.

A topic supports independent subscribers, but it makes the event contract long-lived. Adding subscribers is easy only when the event name and schema are stable enough.

Pub/sub filtering reduces noise, but routing rules can become a second place where business meaning hides. If a rule says "send premium orders to this consumer," ask who owns the definition of premium.

A log supports replay and backfill, but replay is not free. Consumers may redo side effects unless they are idempotent. Retention costs storage. Partitioning and ordering choices become visible design constraints.

This does not solve delivery semantics by itself. A topic does not promise exactly-once side effects. A log does not guarantee that replay is safe. A queue does not prove the worker succeeded. Later lessons cover ordering, idempotency, retries, and delivery semantics. Here the boundary is narrower: choose the shape that matches the consumer relationship.

Signals to watch:

• queue depth and age of oldest message for work queues
• per-subscription lag for topics and pub/sub
• consumer offset lag and retention headroom for logs
• duplicate side-effect or idempotency conflict counts during replay
• messages that are routed to no subscription or to the wrong subscription

Common Confusions

Confusion: "A queue is the generic word for all messaging"

Why it is tempting:

Teams often say "queue" when they mean "some asynchronous place to put messages."

Better model:

Use "queue" when messages are competing work items. Use "topic" or "pub/sub" when independent subscribers should each receive the fact. Use "log" when durable history and replay matter.

Confusion: "A topic means every worker gets every message"

Why it is tempting:

The word "broadcast" can make it sound like every process should receive a copy.

Better model:

Each independent subscription should receive the event. Inside one subscription, a worker pool may still share work so one worker handles one event for that consumer.

Confusion: "A log is always the most correct architecture"

Why it is tempting:

Logs support replay, backfill, and independent offsets, so they sound strictly better.

Better model:

A log buys history by adding retention, offset, replay, schema, and ordering responsibilities. Use it when the history is a product or recovery requirement, not because it sounds more advanced.

Design Review

Use this review when choosing a topology.

Check: Analytics discovers that yesterday's order report counted refunds incorrectly. It needs to rebuild a derived table from yesterday's order events. Which topology feature matters most?

Think first, then reveal.

Answer: Retained history with an independent read position matters most. A log or log-like retained topic lets analytics replay the event range without asking the order service to resend facts or blocking other consumers.

Practice

Review this proposed design:

Product service publishes ProductPriceChanged.

Consumers:
- Search index updates displayed prices.
- Recommendation system recalculates offers overnight.
- Cache invalidation workers remove stale product pages.
- Audit team may need to reconstruct all price changes for the last 90 days.

Choose topology shapes for the consumers. Do not name a vendor product. Name the consumer relationship.

A strong answer should include:

• which consumers need every price change
• which work can be shared by a worker pool
• whether replay or retention is required
• one signal that would show each path is unhealthy
• one boundary that should stay out of this lesson and move to deeper broker internals

Model answer:

ProductPriceChanged is a fact owned by the product service, so independent consumers should see it through a topic or pub/sub channel. Search needs every relevant price change, but its indexing workers can compete inside the search subscription. Cache invalidation also needs every relevant price change, with workers sharing invalidation tasks. Recommendations and audit both benefit from retained history. Recommendations may replay a recent range for nightly rebuilds; audit needs a stronger retention requirement such as 90 days.

Useful signals include search subscription lag, cache invalidation queue age, recommendation consumer offset lag, and retention headroom for audit. The exact partition implementation, replication strategy, broker storage engine, and consumer group protocol belong in the deeper messaging internals track.

Resources

• [BOOK] Martin Kleppmann - Designing Data-Intensive Applications
  • Link: https://dataintensive.net/
  • Focus: Read the chapters on streams, messaging, logs, and derived data for the systems model behind topology choices.
• [ARTICLE] Martin Fowler - What do you mean by Event-Driven?
  • Link: https://martinfowler.com/articles/201701-event-driven.html
  • Focus: Use the distinctions between event notification, event-carried state transfer, event sourcing, and CQRS to avoid treating every event shape as the same.
• [DOC] Microsoft Azure Architecture Center - Publisher-Subscriber pattern
  • Link: https://learn.microsoft.com/en-us/azure/architecture/patterns/publisher-subscriber
  • Focus: Review the basic producer, broker, and subscriber shape and the operational cautions around asynchronous delivery.
• [DOC] Amazon Web Services - Using Amazon SQS and Amazon SNS together
  • Link: https://docs.aws.amazon.com/sns/latest/dg/sns-sqs-as-subscriber.html
  • Focus: Use it only as a concrete example of topic fanout into per-consumer queues, not as the required product choice.

Key Takeaways

• Choose topology from the consumer relationship: shared work, independent observation, filtered subscription, or durable history.
• A queue is good when one worker from a pool should handle one task.
• A topic or pub/sub shape is good when several independent subscribers must each receive the same fact.
• A log is useful when consumers need retained history, independent offsets, replay, or backfill.
• Topology does not solve delivery semantics, idempotency, ordering, or replay safety by itself; it only chooses the movement shape that makes those responsibilities visible.