Serialization Formats and Binary Contracts

Learning / Data Systems / Data Architecture and Platforms / 011

LESSON

Data Architecture and Platforms

011 30 min advanced

Day 459: Serialization Formats and Binary Contracts

The core idea: A serialization format is not just a way to pack bytes; it is a contract for how every producer and consumer identifies fields, represents values, and safely ignores what it does not understand.

Today's "Aha!" Moment

In Declarative Queries and Execution Thinking, PayLedger used indexes and plan shape to find the next payroll runs that are ready to settle. That work happens inside one database engine. The moment those rows leave the engine, the problem changes. A Java settlement coordinator has to send them to a Go bank adapter over gRPC, publish a SettlementReady event to Kafka for downstream reconciliation, and expose a subset of the same facts to a browser-based operations console. At that point the system is no longer moving rows. It is moving bytes across process, language, and version boundaries.

That is where serialization stops being plumbing. If PayLedger encodes amount_minor, currency, settlement_deadline_unix_ms, and trace_id as JSON for every boundary, the messages are easy to inspect but larger and more expensive to parse. If it switches to Protobuf or Avro, the payloads get tighter and parsing usually gets cheaper, but only if the field identity rules, numeric representations, and optional-value semantics are explicit. Binary does not rescue an ambiguous contract. It makes ambiguity harder to see.

The misconception to drop is that a "binary format" is mainly a performance trick. In production, it is first a coordination mechanism. The crucial question is not "can this object be serialized?" but "will every reader, written by different teams in different languages and upgraded at different times, derive the same meaning from the same bytes?" Once you frame the problem that way, wire format choice becomes an architectural decision about correctness, operability, and evolution, not just bandwidth.

Why This Matters

PayLedger runs close to settlement cutoffs. Every minute matters because approved payroll runs have to become bank instructions before a local clearing window closes. The coordinator queries for the next fifty ready runs, then emits a SettlementInstruction to the internal bank adapter and a matching event to the audit stream. If serialization is sloppy, the failures do not look like syntax errors. They look like money moving late, retries piling up, and operators staring at two services that each claim they processed "the same" instruction.

A lot can go wrong at this layer. A Java service may serialize a decimal amount as 12500.00, while a JavaScript consumer rounds it or treats it as a floating-point number with the wrong unit. A hand-rolled binary struct may assume little-endian 64-bit integers, then break when another consumer reads the stream on a different runtime or inserts a new field in the middle. A text-heavy format may spend more CPU parsing field names than the business logic spends validating the instruction. None of those problems are fixed by adding another retry.

When teams handle serialization deliberately, the boundary becomes predictable. They decide whether money is always carried as integer minor units, whether timestamps are always UTC milliseconds since epoch or RFC 3339 strings, whether unknown fields can be skipped, and whether the same semantic event needs different encodings for different audiences. That discipline is what makes later compatibility work possible, which is why the next lesson on Schema Evolution and Compatibility Modes builds directly on this one.

Learning Objectives

By the end of this session, you will be able to:

  1. Explain what a binary contract includes - Go beyond "bytes on the wire" to field identity, value representation, optionality, and decoding rules.
  2. Trace how one message crosses boundaries - Follow PayLedger's SettlementInstruction from in-memory object to API payload, RPC frame, and event record.
  3. Choose a format for the right boundary - Match JSON and binary formats to interoperability, performance, and operational constraints instead of treating one format as universally best.

Core Concepts Explained

Concept 1: Serialization defines a portable meaning, not a dump of memory

Inside the settlement coordinator, SettlementInstruction is just an in-memory object. The Java heap, the Go runtime, and the TypeScript frontend all store values differently. Field order in memory can change. Integers may have different default widths. Structs may include padding. Strings may be represented as references, slices, or UTF-16 code units. None of that matters until the object has to leave the process. Then the system needs a portable representation that every participant can decode the same way.

For PayLedger, the instruction has a few fields that sound simple but are semantically sharp:

run_id
employer_id
amount_minor
currency
settlement_deadline_unix_ms
trace_id

A real wire contract has to answer concrete questions about each of them. How is a field identified: by name, by numeric tag, or by fixed position? How is an integer represented: as text digits, fixed-width binary, or variable-length binary? How is missing data represented: absent, null, or a default value? Are timestamps always UTC? Is amount_minor guaranteed to be cents, not euros and not floating point? Those are not implementation details. They are the meaning of the message.

This is why dumping raw struct bytes is almost never a serious cross-service contract. Raw memory layout reflects compiler, runtime, alignment, and machine assumptions. A durable wire contract has to be independent of those assumptions. Text formats such as JSON make that separation obvious because the serialized form is visibly different from memory. Binary formats do the same job more compactly, but they still need the contract to be designed explicitly.

The trade-off begins here. Self-describing text formats repeat field names and are easy to inspect in logs and debugging tools. Schema-driven binary formats remove repetition and parse more efficiently, but they push more responsibility onto field numbering, code generation, shared schemas, or registries. The format choice is therefore really a choice about how much meaning lives inside each message versus beside it.

Concept 2: Binary contracts work because readers can recover field boundaries and types

Suppose PayLedger uses Protobuf for the gRPC call from the coordinator to the bank adapter:

message SettlementInstruction {
  string run_id = 1;
  int64 employer_id = 2;
  int64 amount_minor = 3;
  string currency = 4;
  int64 settlement_deadline_unix_ms = 5;
  string trace_id = 6;
}

The important property is not just that this is "binary." It is that each encoded field carries enough information for a decoder to recover the boundary and interpret the value. In tagged formats like Protobuf, the wire stream contains field numbers and wire types, followed by the bytes for each value. A simplified mental model looks like this:

[field 1 tag][len][run_id bytes]
[field 2 tag][varint employer_id]
[field 3 tag][varint amount_minor]
[field 4 tag][len][currency bytes]
[field 5 tag][varint settlement_deadline_unix_ms]
[field 6 tag][len][trace_id bytes]

That structure matters operationally. A decoder that does not know about field 6 can still skip it because the tag and wire type tell the decoder how many bytes belong to that field. Small integers consume fewer bytes because varints compress low numeric values. Strings and embedded messages carry lengths so the stream stays parseable. The contract is compact, but it is still structured enough to survive across languages and versions.

Not all binary formats make the same trade. Avro usually depends more heavily on an external writer schema and reader schema instead of carrying field tags in every record. That reduces per-record overhead, which is attractive for event streams, but it means schema distribution becomes part of the runtime contract. A positional binary layout goes further in the dangerous direction: if a record is just "8 bytes amount, then 8 bytes deadline, then 3 bytes currency," inserting one field or changing a width shifts everything after it. The payload may still be small, but it is brittle because readers cannot re-synchronize safely once the contract drifts.

This is also where semantic choices matter more than encoding efficiency. If PayLedger encodes money as a float because it is convenient in one language, the binary payload can be perfectly well-formed and still be wrong for accounting. If timestamps are local-time strings with no zone, both JSON and Protobuf can transport them faithfully and still leave consumers unable to agree on when settlement actually closes. Serialization is therefore about exactness of representation before it is about compactness.

Concept 3: Good systems pick formats per boundary and keep the semantics stable

PayLedger should not force one serialization format onto every boundary just for uniformity. The public partner-facing API has different pressures from the internal settlement RPC path, and both differ from the audit event stream. A useful rule is to choose the format that makes the dominant risk cheapest to manage.

For a public or partner API, JSON is often still the right choice. Tooling is universal, browser clients understand it directly, and humans can inspect requests during incidents without special decoders. That convenience is worth the extra bytes when the boundary is broad and externally integrated. But the team still has to define semantics tightly: money should usually travel as a string decimal or integer minor units, timestamps should carry an explicit timezone, and enums should be documented as stable values rather than incidental strings from one codebase.

For the internal gRPC call between the coordinator and bank adapter, Protobuf is a stronger fit. PayLedger controls both ends, needs low latency under burst load, and benefits from generated clients that keep field types aligned. The compact encoding reduces payload size, while tagged fields give the team a safer base for change. For the Kafka audit stream, Avro or Protobuf with an explicit schema-governance story can both work; what matters is that the event contract is treated as a long-lived data product, not as a transient object dump from one service.

The production trade-off is that format diversity adds translation work. The same SettlementInstruction may be rendered as JSON at the edge, Protobuf on RPC, and a governed event schema in the log. That sounds messy until you compare it with the alternative: forcing one format into boundaries where it is a poor fit, then compensating with custom parsers, ad hoc conventions, or repeated firefighting. Stable semantics with intentional boundary-specific encodings is usually the cleaner system. The next lesson extends that idea by asking how to change those contracts safely once producers and consumers ship on different timelines.

Troubleshooting

Issue: A binary migration cuts network bytes, but end-to-end latency barely improves.

Why it happens / is confusing: Serialization may not be the dominant cost. Compression, TLS, queueing, database lookups, and downstream service time can still dominate the critical path. Teams also overestimate the benefit of binary encoding when large strings or nested payloads still make up most of the message.

Clarification / Fix: Profile serialization and deserialization directly, inspect message shapes, and keep hot-path messages narrow. Treat a format migration as one lever in the path, not a guaranteed latency cure.

Issue: A consumer starts reading nonsense values after another team "just added one field."

Why it happens / is confusing: The real contract may have been positional or coupled to local struct layout even if nobody wrote that rule down. Once offsets shift or field widths change, the old reader has no reliable way to recover where the next value begins.

Clarification / Fix: Use a format with explicit field boundaries and stable identity, and treat field numbers or schema definitions as part of the production API. Never assume local memory layout is the wire format.

Issue: Money totals or deadlines drift after round trips between services.

Why it happens / is confusing: The payload can be syntactically valid while the semantic representation is wrong. Floats introduce rounding risk for money, and ambiguous timestamp strings let consumers interpret the same wall-clock value differently.

Clarification / Fix: Encode money in integer minor units or a clearly specified decimal representation, encode time in an explicitly zoned format, and document those rules in the schema or contract notes instead of relying on caller intuition.

Advanced Connections

Connection 1: Query planning and serialization are one pipeline

This lesson follows Declarative Queries and Execution Thinking on purpose. If PayLedger fetches too many columns or materializes large intermediate rows, it pays that cost twice: once in execution and again when the result is serialized for RPC or events. Narrow projections and clear field semantics make both the database plan and the wire contract cheaper.

Connection 2: Binary contracts are the substrate for schema evolution

Schema Evolution and Compatibility Modes picks up at the exact point where this lesson stops. Once the team has chosen tags, field names, defaults, and message boundaries, it still has to answer harder questions: which changes are backward compatible, which consumers may lag behind, and how compatibility is enforced in CI or a schema registry. You cannot reason about safe evolution until the underlying wire contract is explicit.

Resources

Key Insights

  1. Serialization is a semantic contract - The hard part is not turning values into bytes; it is ensuring every producer and consumer agrees on field identity, units, optionality, and time semantics.
  2. Binary safety comes from structure, not opacity - Tagged fields, lengths, and schema rules are what make compact encodings reliable across languages and versions.
  3. Format choice belongs to the boundary - Public APIs, internal RPC, and durable event streams have different dominant risks, so the best serialization format is the one that makes that boundary easiest to operate correctly.
PREVIOUS Declarative Queries and Execution Thinking NEXT Schema Evolution and Compatibility Modes

← Back to Data Architecture and Platforms

← Back to Learning Hub