Serialization, Schemas, and Protocol Choices

By the end of this lesson, you will be able to...

• Explain why serialization is a boundary contract, not just object-to-bytes conversion.

• Compare JSON, Protocol Buffers, and Avro from boundary constraints instead of format preference.

• Review a schema change for compatibility risks across clients, services, and event consumers.

Idea in one sentence: A wire format is a promise about meaning across time, language, deployment, and ownership boundaries.

Core Insight

In the previous lesson, the learning platform had to decide which layer knew enough to retry a request safely. Now the same platform has a different boundary problem.

The progress service records that learner 7 completed lesson 041. That fact must travel to several places:

• the mobile app, so the learner can see the lesson as complete
• the recommendation service, so it can choose the next lesson
• the certificate service, so it can update completion requirements
• the analytics pipeline, so reports can count progress later

The naive idea is:

We just need to serialize the object and send it.

That sounds reasonable. Inside one process, the progress object is ordinary local data. Across a network or log boundary, it becomes a message that other systems must understand without seeing the original code, memory layout, database model, or release history.

Serialization is the moment local meaning becomes a cross-boundary contract.

The hard part is not only "can the receiver parse these bytes?" The harder questions are:

• What does each field mean?
• Which fields are required?
• What should an old reader do with a new field?
• What should a new reader do with an old message?
• How long can this message live?
• Who is allowed to change the contract?

Those are design questions. JSON, Protocol Buffers, Avro, and other formats are tools for answering them under different constraints.

The Small Situation

Start with one local object in the progress service:

ProgressUpdate
  learner_id: "7"
  lesson_id: "041"
  percent_complete: 100
  completed_at_ms: 1720000000000
  source: "web"

Inside the service, this object is easy to understand because the code gives it context. The field names, types, defaults, and validation rules live nearby.

Now send the same fact to three boundaries:

progress service
  -> public API response for mobile app
  -> internal RPC to recommendation service
  -> event stream for analytics and certificates

The same business fact crosses three different boundaries. Each boundary has a different audience and a different failure cost.

The mobile app may be old for weeks. The recommendation service may deploy several times per day. The analytics job may replay messages written months ago. A contract that is fine for one boundary may be risky at another.

Plain meaning:

A boundary contract is the agreement a receiver depends on when it cannot see the sender's local code.

In this scenario:

The mobile app does not know the progress service's internal model. It only sees the response payload. The analytics job does not know today's service code when it replays last month's event. It only sees stored messages.

Technical name:

The encoded message plus its compatibility rules form a serialization contract.

Where The Naive Design Breaks

Suppose the first version uses a loose JSON payload:

{
  "learner_id": "7",
  "lesson_id": "041",
  "percent_complete": 100,
  "completed_at": 1720000000000
}

This works on day one. It is readable. It is easy to log. The mobile app can parse it with standard libraries.

Then the team adds certificates. Certificates need to know whether completion came from a normal lesson page, an admin correction, or a data repair job. A developer adds a field:

{
  "learner_id": "7",
  "lesson_id": "041",
  "percent_complete": 100,
  "completed_at": 1720000000000,
  "completion_source": "web"
}

What happens now?

Some receivers ignore the new field. Some generated clients may reject unknown fields. Some analytics queries may assume the field always exists. A mobile app may keep using the old shape for weeks. A replay job may read old events where completion_source is missing.

The network delivered the bytes. The protocol may have returned a clean 200. The failure is at the meaning boundary: different systems no longer agree on what the message promises.

That is why "just use JSON" and "just use Protobuf" are both incomplete answers. The useful design question is:

Which boundary needs which contract discipline?

A Worked Contract Review

Review the three boundaries for the same progress fact.

Input:
  one progress update from the progress service

Transition:
  the update crosses three boundaries with different readers

Intermediate state:
  mobile app wants inspectable public data
  recommendation service wants fast typed calls
  analytics pipeline wants replay-safe events

Output or decision:
  choose a format and schema policy per boundary

Naive failure contrast:
  one unversioned message shape for every boundary makes easy changes risky

This table does not pick one universal winner. It says the boundary should drive the format.

For the public API, JSON may be a good choice because humans inspect it easily, logs are readable, and client ecosystems are broad. But JSON alone is not the contract. You still need API documentation, compatibility tests, optional field rules, and a clear rule for unknown fields.

For internal RPC, Protocol Buffers may fit because generated clients and typed messages reduce drift. But Protobuf does not remove design judgment. Field numbers become history. Defaults matter. Removing or reusing a field can break old readers.

For event streams, Avro or another schema-oriented event format may fit because messages are stored and replayed. The schema registry or schema history becomes part of the system's memory.

So far:

• serialization turns local data into a boundary contract
• different boundaries have different readers and lifetimes
• a format choice is really a compatibility and operations choice

Check: A public mobile API and an internal high-volume service call both carry ProgressUpdate. Should they automatically use the same wire format?

Think first, then reveal.

Answer: No. They may share the same business meaning, but the boundary pressures differ. The mobile API may value readability and broad compatibility. The internal service call may value generated types, compact messages, and stricter schema discipline.

Schema Evolution Is The Real Test

The first version of a message is rarely the problem. The test comes later, when the system changes.

A schema change is safe only if old and new producers and consumers can overlap without misunderstanding each other. Distributed systems almost always have overlap:

• old mobile apps keep running
• old service instances stay alive during a rolling deploy
• old events remain in logs
• new consumers start reading old data
• test fixtures and data exports preserve old shapes

Consider this Protobuf-style message:

message ProgressUpdate {
  string learner_id = 1;
  string lesson_id = 2;
  int32 percent_complete = 3;
  int64 completed_at_ms = 4;
}

Now the team wants completion_source.

The safe version adds a new field number:

message ProgressUpdate {
  string learner_id = 1;
  string lesson_id = 2;
  int32 percent_complete = 3;
  int64 completed_at_ms = 4;
  string completion_source = 5;
}

Old readers can usually ignore field 5. New readers must still handle messages where field 5 is missing. That means the application needs a default meaning such as "unknown" or a rule that only newer events participate in certificate-source logic.

The unsafe version reuses a field:

message ProgressUpdate {
  string learner_id = 1;
  string lesson_id = 2;
  string completion_source = 3;
  int64 completed_at_ms = 4;
}

Field 3 used to mean percent_complete. Reusing it for completion_source can make old bytes look like a different fact. That is worse than a parse failure because it can create confident wrong meaning.

Plain meaning:

Schema evolution is the discipline of changing a message without breaking readers that are not changing at the same time.

In this scenario:

The progress service can deploy today, the mobile app can update next week, and analytics can replay last month's events. The schema must survive that uneven timeline.

Technical name:

This is backward and forward compatibility. Backward compatibility means new code can read old data. Forward compatibility means old code can tolerate new data.

Choosing A Format From Constraints

A useful format decision starts with questions, not with a favorite technology.

Ask:

1. Who reads this message?
2. How independently do producers and consumers deploy?
3. How long can the message live?
4. How expensive is a misunderstanding?
5. Do humans need to inspect the payload during incidents?
6. Do we need generated clients and strict field IDs?
7. Do we need a schema registry or compatibility gate?

Then compare the options.

This comparison is not absolute. JSON can be used with JSON Schema and strong tests. Protobuf can be misused. Avro can be overkill for a simple endpoint. The format does not save a weak contract.

The design trade-off is speed now versus safety later. A loose shape may help a small team move quickly. A stricter schema slows casual changes but makes independent deployment and replay safer. The right answer depends on the boundary and the cost of being wrong.

Check: An analytics team replays six-month-old progress events into a new report. Which risk should influence the format choice most?

Think first, then reveal.

Answer: Message lifetime and schema evolution. The report reads old data with newer code, so the contract must preserve meaning across time, not only across today's network call.

Common Design Mistakes

Confusion: "Serialization is only performance"

Why it is tempting:

Benchmarks are easy to compare. Payload size and parse speed are visible numbers.

Better model:

Performance matters, but the contract also controls debugging, compatibility, rollout safety, replay safety, and how clearly systems preserve meaning.

Confusion: "Readable means safe"

Why it is tempting:

A JSON payload looks obvious when one team owns both sides and everyone remembers the current shape.

Better model:

Readable is not the same as governed. A readable payload can still have unclear defaults, incompatible type changes, missing-field ambiguity, and undocumented ownership.

Confusion: "Schemas make compatibility automatic"

Why it is tempting:

Generated types feel strict. A build step can make the message look controlled.

Better model:

Schemas give you tools for compatibility. Teams still need rules: do not reuse field IDs, define defaults, reserve removed fields, test old/new combinations, and document semantic changes.

Practice

Review this design proposal.

The progress service wants to publish this event to analytics and certificates:

{
  "learner_id": "7",
  "lesson_id": "041",
  "percent_complete": 100,
  "completed_at": "2026-07-09T10:00:00Z"
}

The team plans three changes next month:

• add completion_source
• rename percent_complete to progress
• let certificates replay all events from the last year

Write a short review with:

1. one safe change
2. one risky change
3. one format or schema-policy recommendation
4. one compatibility test you would require

Model answer:

Adding completion_source as an optional field can be safe if old readers ignore it and new readers define a default for old messages. Renaming percent_complete is risky because old consumers may still depend on that field name, and historical events still use it. For a year-long replay boundary, use an event format with explicit schema evolution rules, or keep JSON only with strict schema checks, versioned contracts, and compatibility tests. A useful test replays old events into the new certificate consumer and sends new events to an old analytics consumer to confirm both sides preserve meaning.

Trade-offs and Limits

Explicit contracts improve independent deployment. They make it easier for old and new producers, clients, services, and consumers to overlap safely.

They cost time and process. Someone must own schema review. Compatibility tests must run. Developers must learn which changes are safe, risky, or forbidden. A schema registry, generated clients, or API contract test suite is useful only if teams respect it.

This lesson helps with boundary format decisions. It does not teach full protocol internals, binary encoding details, HTTP semantics, or storage design for event logs. Those topics live in deeper tracks.

You can see the boundary when a change looks harmless locally but becomes risky across time or ownership. A field rename inside one service is a refactor. A field rename in a public API or replayed event stream is a migration.

Resources

• [ARTICLE] Protocol Buffers Language Guide - Study field numbers, unknown fields, defaults, reserved fields, and compatibility discipline.
• [ARTICLE] The JavaScript Object Notation Data Interchange Format - Revisit why JSON is broadly interoperable and human-friendly despite weak built-in schema rules.
• [ARTICLE] Apache Avro Specification - Compare a schema-oriented format designed for long-lived data exchange and schema resolution.
• [BOOK] Designing Data-Intensive Applications - Use the encoding and evolution chapters to connect message formats with independent deployment.

Key Takeaways

• Serialization preserves shared meaning across boundaries, not just bytes on the wire.
• Format choice should follow the boundary: public API, internal RPC, event stream, or long-lived data pipeline.
• Schema evolution is where the contract proves whether independently deployed systems can still understand each other.
• Readability, efficiency, tooling discipline, and compatibility safety are trade-offs, not separate checkboxes.
• A field change can be a local refactor inside one process and a breaking migration across a network or replay boundary.