Schemas, Contracts, and Versioned Messages

By the end of this lesson, you will be able to...

• trace a message change across old producers, new producers, old consumers, new consumers, and retained messages.

• distinguish schema shape compatibility from semantic compatibility.

• design a safe rollout and retirement boundary for a versioned message contract.

Idea in one sentence: A message contract is safe to change only when every participant that may still exist can parse the message and interpret its meaning correctly.

Core Insight

A digital library has a service that records ebook loans.

When a reader borrows a book, the library service publishes this event:

loan_started
  loan_id
  user_id
  book_id
  due_at

Several services consume it.

email service        -> send a confirmation
recommendation index -> update "recently borrowed"
late-fee service     -> schedule reminders
analytics pipeline   -> count loans by day

Now the product team wants family accounts. A child can borrow a book through a parent account. The team wants to add a borrower type:

borrower_type: adult | child

That sounds like a small field. But the system is distributed. Not every service deploys at the same time. Some old loan_started events are still in queues. Some events may be replayed next month to rebuild the recommendation index.

If the new library service publishes a message that an old consumer rejects, the rollout breaks. If a new consumer assumes every old event has borrower_type, old messages break. If a team silently changes the meaning of user_id, the message may parse while the business behavior becomes wrong.

Plain meaning:

A contract is the agreement that says, "When this message appears, this is its shape, this is what each field means, and these older and newer participants can safely coexist."

In this scenario:

The library service, email service, recommendation index, late-fee service, analytics pipeline, queued messages, and replay tools all need a safe interpretation of loan_started.

Technical name:

The message shape is a schema. The agreement about shape, meaning, compatibility, rollout, and retirement is a message contract.

The Naive Idea: Add The Field And Deploy

The first plan is tempting:

1. Add borrower_type to loan_started.
2. Deploy the library service.
3. Update consumers later.

That might work in a single process. It is risky in a distributed system.

At 10:00, the library service deploys.

new producer writes:
loan_started v2
  loan_id: loan-8
  user_id: user-parent-3
  book_id: book-44
  due_at: 2026-07-15
  borrower_type: child

At 10:05, the email service is still old. It expects only v1 fields.

There are several possible outcomes:

• it ignores the unknown field and sends the email;
• it rejects the message because the schema is unknown;
• it sends the email but uses the wrong wording;
• it crashes and blocks the queue.

The important question is not "Did the producer deploy cleanly?"

The important question is:

Which old and new readers, writers, and stored messages must coexist?

Check: Why is "the new producer can write the new field" not enough to prove the rollout is safe?

Think first, then reveal.

Answer: Because the producer is only one participant. Old consumers may still read new messages, new consumers may still read old messages, and retained messages may reappear through queues, retries, dead-letter stores, or replay.

Shape And Meaning Are Different Contracts

A schema usually describes shape.

field name: borrower_type
field type: string or enum
required?: no
allowed values: adult, child

Shape compatibility asks:

Can this reader parse this message?

Shape matters. Removing a required field, changing a number into a string, or adding a required field without a default can break readers immediately.

But shape is not enough.

Imagine this change:

v1 meaning:
  user_id = the account that receives the loan

v2 meaning:
  user_id = the person who physically reads the book

The field name did not change. The field type did not change. A schema checker may be happy.

The late-fee service may not be happy.

If user_id used to identify the billing account and now identifies the child reader, reminders, limits, and permissions may go to the wrong place. The message parses, but the meaning changed under the same label.

Semantic compatibility asks:

Will old and new participants interpret the message the same safe way?

For this feature, a safer design is to preserve the old meaning and add a new field:

loan_started v2:
  loan_id
  account_user_id      # old user_id meaning, renamed only when safe
  borrower_profile_id  # new reader identity
  borrower_type
  book_id
  due_at

If the old name cannot be removed yet, keep it with its old meaning and document the new field clearly:

user_id = account that owns the loan
borrower_profile_id = profile that will read the book

That is less tidy than a quick rename. It is safer because readers do not have to guess which meaning is hiding behind a familiar field.

Compatibility Has Direction

Teams often say, "This schema is compatible."

That sentence is incomplete. Compatible in which direction?

Backward compatibility means new readers can read old messages.

In the library example:

new late-fee service reads old v1 message without borrower_type

The new service needs a documented behavior:

if borrower_type is missing:
    treat as adult account loan
    or look up borrower profile from durable loan record
    or send to repair if the meaning is unsafe

The answer depends on the product truth. A default is safe only if it is true for old data.

Forward compatibility means old readers can tolerate new messages.

old email service reads v2 message with borrower_type

This is safe only if the old email service ignores unknown optional fields or uses a decoder that preserves them without rejecting the message.

Full compatibility means both directions hold during the mixed fleet:

new reader + old message -> safe
old reader + new message -> safe
new reader + new message -> safe
old reader + old message -> still safe

Version numbers help, but they are not the contract by themselves. A field called schema_version = 2 is useful only if readers know what version 2 means, which old versions may still appear, and when old support can be removed.

A Worked Trace: Add Borrower Type Safely

Follow the rollout as a coexistence problem.

1. Input: The Desired Change

The product wants child profiles.

Naive change:

loan_started v2:
  loan_id
  user_id
  book_id
  due_at
  borrower_type

Risk:

Some consumers still run v1 code. Some old v1 messages still exist. Replay tools may read both versions later.

2. Transition: Prepare Readers First

Before the producer emits v2, deploy readers that can handle both forms.

email service v2:
  v1 message -> send normal adult/account email
  v2 adult  -> send normal adult/account email
  v2 child  -> send parent-account wording

late-fee service v2:
  v1 message -> use account-level policy
  v2 message -> use borrower_type if present

Intermediate state:

producer: old
some consumers: new
messages: old shape only

This state is boring. Boring is good here. New readers are ready before new messages arrive.

3. Transition: Emit The New Optional Field

Now the producer starts sending borrower_type.

producer: new
consumers: mixed
messages: old and new

The compatibility matrix must still work.

old email + v2 message -> ignores borrower_type, still sends safe generic email
new email + v1 message -> uses old account behavior
new email + v2 message -> uses borrower-specific behavior

Output:

Family loans can begin, but the rollout does not require every consumer to switch behavior at the same moment.

Naive failure contrast:

If the producer had made borrower_type required before readers were ready, old messages would fail in new consumers. If old consumers rejected unknown fields, new messages would block old consumers. The small field would become a deployment incident.

4. Transition: Depend On The Field Only After Evidence

The team should not immediately write product rules that require borrower_type everywhere. First it checks evidence:

oldest queued loan_started version
percentage of consumers upgraded
schema validation failures
missing-field fallback count
dead-letter count by schema version
replay test result

Only after old producers are gone, queues have drained, and replay behavior is tested can the new behavior depend on the new field.

5. Output: Retire Old Support Deliberately

Removal is part of the contract.

retire v1 reader path only after:
  no v1 messages remain in live queues
  retained event policy says v1 will not be replayed without translation
  repair and replay tools can handle v1
  consumers no longer report missing borrower_type fallback

Until then, compatibility code is not clutter. It is part of the system's ability to recover from old data.

Check: A new consumer reads an old loan_started message without borrower_type. Is that a forward-compatibility or backward-compatibility question?

Think first, then reveal.

Answer: Backward compatibility. New code is reading old data. The consumer needs a safe behavior for messages produced before the new field existed.

Replay Makes Old Contracts Current Again

Many message bugs wait until replay.

The recommendation index may be rebuilt from last month's loan_started events. That replay can send old v1 events through today's code. If today's code assumes every message has borrower_type, the bug appears long after the deployment looked successful.

A good rollout tests replay explicitly:

replay sample:
  v1 adult loan -> current recommendation consumer
  v2 adult loan -> current recommendation consumer
  v2 child loan -> current recommendation consumer
  malformed message -> visible dead-letter and repair path

Replay also tests semantics. The consumer should not merely parse the old message. It should produce the documented old meaning. If old user_id means account owner, replay should not quietly reinterpret it as child profile.

This is where contracts become operational evidence. The team should know:

• which schema versions are still stored;
• which consumers still use fallback paths;
• which messages fail validation;
• which dead-letter messages need repair;
• which replay tools can translate old events safely.

Common Confusions

Confusion: Optional Means Harmless

Why it is tempting:

An optional field does not force every producer to send it.

Better model:

Optional only solves one shape problem. The field can still change meaning, be rejected by old readers, be required too early by new readers, or be missing from replayed data.

Confusion: A Schema Registry Owns The Contract

Why it is tempting:

Schema registries can enforce useful compatibility checks.

Better model:

A registry can check many shape rules. It cannot know every product meaning. It cannot decide whether user_id means account owner or child profile unless humans make that contract explicit.

Confusion: Version 2 Replaces Version 1 Immediately

Why it is tempting:

After the deploy, the current code writes version 2.

Better model:

Old messages can survive in queues, logs, backups, dead-letter stores, and event archives. Version 1 is gone only when every retention and replay path says it is gone or translatable.

Trade-offs And Limits

Safe message evolution trades speed for independent deployment.

The fast path is one big change: update the producer, update all consumers, rename fields, delete old code, and hope the rollout is perfectly synchronized.

The safer path is slower:

• define the field meaning;
• prepare readers before writers depend on the new shape;
• keep temporary compatibility code;
• monitor old and new message versions;
• retire old behavior only after retention and replay boundaries are clear.

The trade-off is real. Compatibility code adds branches, tests, documentation, and cleanup work. It can become permanent if nobody owns retirement.

But the payoff is large: services can deploy independently, queues can drain safely, and incident recovery can replay old data without turning old messages into new failures.

This idea does not solve every data problem. A compatible schema can still carry wrong facts. A well-versioned message can still be processed twice unless the consumer is idempotent. A safe rollout can still fail if a hidden consumer exists outside the known contract.

You can see the boundary when validation succeeds but business behavior is wrong. That is the signal to review semantic meaning, not just schema shape.

Another boundary is ownership. A contract without an owner tends to decay. New consumers appear, old replay jobs keep running, and nobody knows which version can still be removed. Versioned messages need a named owner for compatibility decisions, not just a file in a repository.

Practice: Review A Message Change

Review this proposed change:

message: loan_started
current fields:
  loan_id
  user_id
  book_id
  due_at

proposed change:
  add borrower_type
  start using user_id as the borrower profile id

Fill in:

old consumer reading new message:
new consumer reading old message:
shape risk:
semantic risk:
safe rollout order:
retirement evidence:

Model answer:

old consumer reading new message:
  must ignore borrower_type or remain on v1 until it can tolerate it

new consumer reading old message:
  must handle missing borrower_type with a documented safe behavior

shape risk:
  old readers may reject unknown fields; new readers may require borrower_type too early

semantic risk:
  changing user_id from account owner to borrower profile silently breaks old meaning

safe rollout order:
  preserve user_id meaning, add borrower_profile_id and borrower_type, deploy tolerant readers, then emit new fields

retirement evidence:
  no live v1 messages, replay path tested, fallback count near zero, owner approves old support removal

Resources

• [REFERENCE] Apache Avro: Schema Resolution - Focus: How readers and writers with different schemas resolve fields and defaults.
• [ARTICLE] Confluent Schema Registry: Schema Evolution and Compatibility - Focus: Backward, forward, and full compatibility rules for event schemas.
• [BOOK] Designing Data-Intensive Applications - Focus: Schema evolution, data encoding, and compatibility across independently deployed systems.

Key Takeaways

• A schema describes message shape; a contract also includes meaning, rollout, retention, and retirement.
• Compatibility has direction: new readers reading old messages is different from old readers reading new messages.
• Safe rollout usually prepares tolerant readers before producers depend on a new message shape.
• Semantic compatibility can break even when the message parses correctly.
• Old messages remain relevant while queues, archives, dead-letter stores, and replay tools can still surface them.