Networks and Distributed Reality

By the end of this lesson, you will be able to...

• trace a request across a client, a network, and a remote service;

• explain why a timeout does not reveal whether the remote work happened;

• distinguish a retry from a safe retry using a stable request identity;

• identify latency, partial knowledge, and duplicate work as network pressures.

Idea in one sentence: A remote request is not a local function call with extra delay: messages can be late, lost, duplicated, or answered after the caller has given up.

Core Insight

Suppose Nora presses “Pay €20” in a shop app. The app sends a request to a payment service. After two seconds, no answer arrives, so the screen shows “Try again.” Nora presses the button a second time.

What happened to the first request? From the app’s point of view, several stories fit the same timeout:

• the request never left the phone;
• it reached the payment service, which never processed it;
• the payment service charged the card, but its reply was lost or delayed;
• the payment service is still deciding when the app starts the retry.

This is the central change introduced by a network. Each participant sees only its own events and the messages it has received. The caller cannot inspect a remote machine’s current state just because it sent a message.

Networks give programs reach: the app can use a payment service, a database, a notification provider, or a machine on another continent. The trade-off is that this reach removes local assumptions. A function call in one process usually has one stack, one memory space, and a direct return or exception. A remote request crosses independent machines and an unreliable path between them.

The Small Situation

Keep the scenario deliberately small. There are three actors:

Nora's app  ->  network  ->  payment service

The app sends a charge request containing an amount and an order identifier. The payment service receives a request, asks the card processor to charge, records the result, and sends a response.

The product promise is simple: one order should create at most one charge. The mechanism is not simple because the app and payment service may disagree temporarily about whether the first attempt finished.

The naive model is familiar:

result = charge_card(order, €20)
if result is success:
    show confirmation
else:
    show error

This model works well when charge_card is local. The caller waits for a result from the same process. With a remote service, “no result yet” is different from “the work did not happen.”

The Moving Parts

Before tracing the request, name what each part can know and do.

The network is not a conscious component choosing to be difficult. It is a collection of links, queues, devices, protocols, and failure boundaries. The table is a reasoning tool: it stops us from giving one participant knowledge that belongs to another.

The Mechanism Step by Step

Here is one possible timeline for the first press:

Input: Nora’s click creates request p-81.

Transition: the request reaches the payment service, which records a successful charge.

Intermediate state: the service has a durable answer, but the app has no answer. This is partial knowledge: the two sides hold different information because the reply has not reached one of them.

Output or decision: the app sees a timeout. A timeout is evidence that its waiting period ended, not proof that the charge failed.

Naive failure contrast: if the app creates a fresh charge request on every retry, the second press may produce two charges. The local-function intuition—“an exception means nothing happened”—does not survive this boundary.

A Better Request Contract

The first improvement is not “retry forever.” It is to give the operation a stable identity. The app creates one idempotency key, p-81, for Nora’s intended payment and reuses it for retries of that same intent.

charge(order-44, €20, idempotency_key = p-81)

The payment service stores the outcome by key:

if p-81 already has a recorded result:
    return that result
otherwise:
    perform the charge
    record its result under p-81
    return the result

Plain meaning:

Several deliveries of the same request should have the effect of one intended payment, not several payments.

In this scenario:

The second press can resend p-81. The service finds the recorded success and returns it instead of charging again.

Technical name:

This property is idempotency. Repeating an operation with the same identity has the same intended effect as doing it once. It is not a magical network guarantee; it is a contract the service implements by recognizing repeated intent.

A Worked Retry Trace

Now trace the retry with the same key.

App                         Payment service
 | charge(..., p-81)  ----> |
 |                          | charge succeeds; record p-81 = success
 | <---- success reply -----|  (reply is lost)
 | timeout                  |
 | charge(..., p-81)  ----> |
 |                          | find p-81 = success
 | <---- success reply -----|
 | show one confirmation    |

The retry changes the service’s decision. It does not need to infer whether the first reply reached the app. It only needs to recognize that both arrivals represent the same payment intent.

There is still an important boundary. The service must record the result reliably enough that a crash between “charge” and “record result” does not turn a retry into a second charge. In real payment systems, this can require cooperation with the processor, transaction design, and reconciliation. The idempotency key reduces a major risk; it does not make all failure modes disappear.

So far, the key model is: a request can be delivered zero or more times, and a reply can be absent even after remote work succeeds. Good contracts make repeated intent safe and make uncertain outcomes visible.

Latency, Ordering, and Identity

Latency is the time before a message or reply arrives. It is not merely an annoyance. A slower response means the app has more time to make a wrong assumption, show an uncertain state, or send overlapping work.

Ordering is also not automatic at the product level. Suppose Nora changes her delivery address and then pays. If separate services observe those operations in different orders, which address should appear on the receipt? The network may carry messages, but the application must decide whether ordering matters and how to preserve or reconcile it.

Identity lets a system talk about “the same thing” across time and machines: the same order, payment intent, user, request, or event. Without it, a retry is indistinguishable from a new request. With it, a service can deduplicate, return status, and build an audit trail.

These pressures are connected:

• latency creates waiting and timeouts;
• a timeout creates uncertainty;
• uncertainty encourages retries;
• retries require identity to avoid duplicate work;
• multiple operations may require an ordering or coordination rule.

Cost, Limits, and Signals

Remote calls improve reach and separation of responsibility. They cost time, introduce more failure points, and make knowledge uneven. A local check that takes microseconds may become a network dependency that waits seconds during an outage.

Idempotency costs storage for keys and outcomes, a retention policy, and careful definition of what “same request” means. If p-81 is reused accidentally for a different order, deduplication becomes a bug. Keys need a scope and enough uniqueness for the promised behavior.

This lesson also does not say every operation should retry. A request that causes a non-repeatable side effect needs an idempotent contract, a status check, user confirmation, or a different workflow before automatic retry is safe.

Watch the signals:

• rising timeout rates show that callers are losing timely evidence, even if the remote service still succeeds;
• duplicate-key responses show retries are occurring and should be expected, not treated as impossible;
• mismatches between payment records and order records reveal that separate components may have partial or delayed knowledge;
• long queues and high latency can turn an otherwise harmless retry policy into extra load.

Common Confusions

Confusion: “Timeout means failure.”

Why it is tempting: the caller did not receive the answer it wanted.

Better model: a timeout describes the caller’s observation. The remote operation may have failed, succeeded, or still be running. Treat the result as unknown until a contract or status check resolves it.

Confusion: “Retries make a request reliable.”

Why it is tempting: sending again often repairs a lost message.

Better model: a retry can create duplicate work. It becomes safer only when the operation is designed to recognize repeated intent or when a duplicate is harmless.

Confusion: “A network is just a slow function call.”

Why it is tempting: both use a request and a response.

Better model: remote participants fail independently and do not share one memory or one clock. Delay changes what each side can know, not only how long it waits.

Check Your Understanding

Check: The payment service records a successful result for p-81, but the app times out. What should the app assume about the charge?

Think first, then reveal.

Answer: It should assume the outcome is unknown from its own point of view. A retry with the same idempotency key or a status query can resolve the uncertainty without automatically creating a new payment intent.

Check: Why is generating a new key for every retry unsafe in this scenario?

Think first, then reveal.

Answer: The service sees different identities and has no basis to recognize them as the same intended payment. If both requests arrive, it may correctly process two distinct-looking charges.

Practice

A notification service sends one password-reset email. The client times out after submitting the request and wants to retry.

Design a small request contract. Specify the request identity, what the service records, what success means to the caller, and one signal that would show a duplicate or delayed outcome. Then decide whether it is acceptable for a user to receive two reset emails, and how that decision changes the retry policy.

A good answer should mention:

• a key tied to one reset intent, not one network attempt;
• a recorded outcome or status the service can return for the same key;
• the difference between accepting the request and delivery to an inbox;
• whether duplicate emails are harmless, confusing, or a security concern, and how logs or metrics would reveal them.

Resources

• [BOOK] Designing Data-Intensive Applications — Focus: Read the chapters on unreliable networks and replication after building this first model.
• [COURSE] MIT 6.5840: Distributed Systems — Focus: Use the introductory material to see how failures and partial knowledge shape system design.
• [REFERENCE] RFC 9293: Transmission Control Protocol — Focus: Notice that even a reliable transport protocol defines delivery machinery, not the product-level meaning of a payment.

Key Takeaways

• A remote timeout tells the caller that it lacks a response, not that the remote work failed.
• Networks create partial knowledge: the app and service can hold different, locally valid views of the same request.
• Stable request identity lets a service recognize retries of one intent and implement idempotent behavior.
• Latency, duplicates, ordering, and independent failure are product and design concerns, not only transport details.
• A good remote contract states what success means, how uncertainty is resolved, and which signals reveal its limits.