Unreliable Networks
The first trap in a distributed system is treating the network like a function call. A local function either returns or throws in the same process. A remote call can leave the caller with a worse problem: it may not know what happened.
Naive Version
send request
wait for response
continue as if response means success
This works while the network is quiet and fast. It breaks when the caller gets a timeout.
A timeout does not prove the remote side did nothing. The request may have been lost. The server may have completed the work but the response was lost. The server may still be processing it. These cases need different recovery behavior, but they look similar from the caller's point of view.
Smallest Improvement
Make the communication state visible.
| Technique | What it makes explicit | Trade-off |
|---|---|---|
| Logging | A call failed or timed out | Helps diagnosis, not recovery by itself |
| Retry with acknowledgement | The caller keeps trying until it gets a known reply | Requires safe duplicate handling |
| Store and forward | Work is durably recorded before delivery | Changes the flow from immediate call to queued delivery |
| Reliable messaging | Transport handles delivery and acknowledgements | Usually does not behave like synchronous request/response |
The important shift is that reliability is built above an unreliable network. That often means giving up the illusion of a simple synchronous call. A queue can make delivery more dependable, but it also changes the user-visible and code-visible model: the caller sends work, then observes completion later.
Design Check
For each remote operation, ask:
- What happens if the request is lost?
- What happens if the response is lost after the work completed?
- Can this operation be retried without doing the work twice?
- Does the caller truly need an immediate answer, or can the system accept queued work?