Network Boundaries, Latency, and Partial Failure

By the end of this lesson, you will be able to...

• trace what each participant can know across a slow network boundary.

• explain why tail latency can turn success into an unknown caller outcome.

• choose honest timeout, deadline, and pending-state behavior for partial failure.

Idea in one sentence: A network boundary turns direct-looking work into message exchange, so latency and partial failure must be handled as evidence problems.

Core Insight

Imagine uploading a profile photo.

You choose an image, press Upload, and wait. The page spins for a few seconds. Then it says:

Upload failed. Try again.

You refresh the page. The photo is there.

That is not just a confusing message. It is a network-boundary story.

The browser knows it waited and did not receive a useful answer. The API gateway may know its timer expired. The media service may know it stored the image object. The database may know it wrote the photo metadata. The thumbnail worker may know nothing yet.

Several true facts exist at once, and none of them is the whole answer.

The naive idea is:

If the upload succeeds, show success.
If the upload fails, show failure.

The network breaks that tidy model. A remote operation can keep running after the caller stops waiting. A response can be lost after the side effect succeeds. One part can be healthy while another part is saturated. The result is not always success or failure. Sometimes the honest result is:

the caller does not know yet

Plain meaning:

The caller and receiver are separated by time, queues, and messages.

In this scenario:

The browser can time out while the media service has already stored the image.

Technical name:

This is partial failure at a network boundary. Some participants keep working while another participant cannot get the evidence it needs.

The Moving Parts

Use this small upload workflow:

browser
  -> gateway
  -> media service
  -> object storage
  -> metadata database
  -> thumbnail worker

Each participant sees a different piece:

browser:
  user selected photo-8.jpg
  upload request started
  no final response yet

gateway:
  request arrived
  request forwarded to media service
  timeout set to 2 seconds

media service:
  received upload
  wrote image bytes to object storage
  wrote metadata row

thumbnail worker:
  may receive work later

The boundary is not only the cable or Wi-Fi. The boundary is the fact that one participant cannot directly inspect or change another participant's local state. It must send a message and wait for evidence.

That evidence may be late, missing, duplicated, or stale.

The Naive Timeline

The happy path is easy:

browser -> gateway: upload photo request
gateway -> media: store photo
media -> object storage: write bytes
object storage -> media: stored
media -> database: write metadata
database -> media: committed
media -> gateway: upload complete
gateway -> browser: saved

The browser sees a clean result. The system has enough evidence to say "saved."

Now add one delay:

time ------------------------------------------------------------>
browser:  send upload ---------------- waits ---------- timeout
gateway:  forward request ------------ waits ---------- timeout
media:          receive -> write object -> write metadata -> respond
network:                                                     response late

Nothing exotic happened. No machine exploded. The media service may have done useful work. The caller still timed out.

That is why a remote call is not a slow function call. With a local function, the caller and callee usually share one process fate. Across a network boundary, the receiver can continue after the caller has stopped waiting.

Check: In the delayed upload timeline, what does the browser timeout prove?

Think first, then reveal.

Answer: It proves that the browser did not receive a useful response before its waiting rule ended. It does not prove that object storage or the database failed.

A Worked Trace

Here is the same upload as a state table.

The input was one user action. The transition was a message crossing the gateway-to-media boundary. The intermediate states matter:

object stored, metadata missing
object stored, metadata committed
caller timed out, owner succeeded
thumbnail not yet generated

The output should match evidence. If the UI says "failed" after step 7, it may be lying. If it says "saved" without checking the owner, it may also be lying. A better state is:

We are checking whether your photo finished uploading.

So far, the key idea is simple: latency creates time for participants to diverge. Partial failure is what that divergence feels like from the outside.

Latency Is A Shape

People often talk as if a dependency has one speed:

media upload takes 300 ms

Real latency has a shape. Most uploads may be fast. Some are slower because of large files, object-storage hiccups, queueing, garbage collection, network retransmits, or overloaded database connections.

A simplified distribution might look like this:

photo metadata write latency

p50:   120 ms   common case
p95:   700 ms   slow but expected
p99:  2400 ms   rare tail request
max:  9000 ms   incident or severe overload

If the gateway timeout is 2000 ms, some p99 requests will cross the timeout even though the media service eventually succeeds.

If the timeout is 9000 ms, fewer successful uploads will become unknown to the caller. But the gateway will keep more requests open during overload. Those waiting requests consume memory, sockets, worker slots, and user patience.

The trade-off is not "short timeout good" or "long timeout good." The trade-off is:

shorter timeout:
  protects caller resources
  creates more unknown outcomes

longer timeout:
  reduces some unknown outcomes
  consumes more resources while waiting

Check: If average upload latency is healthy, can p99 latency still break the user experience?

Think first, then reveal.

Answer: Yes. Users experience individual requests. A rare slow request can cross the caller timeout and create an unknown outcome even when the average looks fine.

Timeouts And Deadlines

A timeout is local:

I will stop waiting after this much time.

A deadline is shared across the operation:

This work is only useful until this time.

Without a deadline, each hop can accidentally restart the clock:

browser waits up to 4 seconds
gateway waits up to 4 seconds for media
media waits up to 4 seconds for object storage
object storage waits on internal work

The user-facing promise may be broken before the last participant notices.

With a deadline, each participant sees the remaining useful time:

browser deadline: 14:03:10.000
gateway receives request at 14:03:06.300
gateway passes deadline to media
media receives request at 14:03:06.450
media knows about 3550 ms remain

The media service can make better decisions. It may skip thumbnail generation, reject early when overloaded, or store the object and mark thumbnail work as pending.

Deadlines do not remove partial failure. They bound the waiting and make the system more honest about what can still finish in time.

Designing The Unknown State

The upload workflow needs more than two states.

Too small:

uploaded
failed

Better:

uploading
stored_object
metadata_committed
thumbnail_pending
outcome_unknown_to_caller
confirmed_uploaded
failed_before_storage
failed_after_storage_needs_cleanup

You do not always expose all of those states to the user. But the system should be able to represent them internally.

A safe policy might be:

if media returns before deadline:
  show uploaded

if gateway timeout fires:
  record upload_outcome_unknown
  ask the media owner about upload-77

if owner says metadata committed:
  show uploaded, maybe thumbnail pending

if owner says no object and no metadata:
  allow retry

if owner cannot answer:
  show checking state and schedule repair

This lesson is not yet the retry lesson. The next lesson handles retries and idempotency directly. Here, the important move is to avoid pretending unknown means failed.

What This Changes

Before this model, it is tempting to design the upload as one blocking action:

call media service
wait
return uploaded or failed

After this model, the upload becomes a small state machine:

start upload
record the operation identity
move through visible intermediate states
ask the owner for evidence after uncertainty
show the user a state that matches what the system knows

That changes product behavior. A timeout no longer has to become a scary final error. The UI can say "checking upload" while the system asks the media owner whether upload-77 exists. Support can inspect the same operation id instead of guessing from one browser error. Operators can look for the boundary where time was spent.

It also changes engineering behavior. The gateway should not own the truth of the uploaded photo. The media service or metadata database should own that fact. The gateway owns the waiting rule and the user-facing response. Those are different responsibilities.

This separation is small, but it prevents a common incident pattern: a caller loses patience, labels the operation failed, and later the receiver proves that the side effect succeeded. Once you expect that shape, you can design the unknown state before production invents it for you.

Operational Signals

Partial failure often hides behind "the service is up."

A health check can pass while the real upload path is unhealthy:

health check:
  gateway -> media /health
  media -> gateway: 200 OK in 8 ms

real upload:
  gateway -> media upload
  media waits for object-storage connection
  gateway timeout fires at 2000 ms

The service is alive. The workflow is not healthy.

Useful signals are tied to the real boundary:

• p95 and p99 latency for upload requests
• timeout count by caller and dependency
• in-flight upload count
• object-storage connection pool saturation
• queue depth for thumbnail work
• late-success count after gateway timeout
• traces that show where time was spent

A useful trace might say:

gateway validation:             20 ms
gateway -> media wait:        2000 ms timeout
media waiting for storage:    1700 ms
object write:                  180 ms
metadata write:                 40 ms
media response after timeout:  120 ms

That trace explains the boundary. The media service was not down. It was waiting for storage long enough that the caller's timeout fired.

Trade-offs and Limits

Timeouts improve resource protection. They stop callers from waiting forever.

They cost certainty. A timed-out caller often needs a later owner check, a pending state, or a repair job.

Deadlines improve budget discipline. They help downstream participants avoid work that can no longer help the user.

They do not guarantee correctness. A service can ignore the deadline, crash after a side effect, or return too late.

Operational signals improve diagnosis. They reveal when a boundary is unhealthy even though individual participants are alive.

They do not replace product policy. The system still needs to decide what to show while evidence is incomplete.

You can see the boundary getting risky when p99 latency approaches the caller timeout, in-flight work rises, and users see ambiguous outcomes.

Common Confusions

Confusion: Timeout means the remote work stopped

Why it is tempting:

In local code, returning from a failed call often feels like the work is over.

Better model:

Across a network boundary, the receiver may keep running after the caller stops waiting.

Confusion: If health checks pass, the workflow is healthy

Why it is tempting:

A green health check is easy to read.

Better model:

The real workflow can be slow or saturated while a cheap health endpoint still returns 200 OK.

Confusion: Longer timeouts always make the system safer

Why it is tempting:

Waiting longer can reduce some unknown outcomes.

Better model:

Longer timeouts also consume caller resources during overload. They can make the system slower for everyone.

Practice

Pick one action that crosses a network boundary: uploading a photo, joining a call, posting a comment, saving a document, or submitting a form.

Write:

caller:
receiver:
owner of the official state:
user-facing deadline:
caller timeout:
what the timeout proves:
what the timeout does not prove:
unknown state:
signal that would reveal partial failure:

Now change one parameter. Cut the caller timeout in half.

Predict:

what improves:
what gets worse:
what state becomes more common:
what signal should you watch:

A good answer should say that shorter waiting protects caller resources but creates more unknown outcomes. It should name an owner check or pending state, not only "try again."

Resources

• [ARTICLE] Timeouts, Retries, and Backoff with Jitter
  • Focus: How timeout choices interact with overload, tail latency, retries, and caller resource exhaustion.
• [ARTICLE] Notes on Distributed Systems for Young Bloods
  • Focus: Practical framing for latency, uncertainty, and defensive network boundaries.
• [BOOK] Designing Data-Intensive Applications
  • Focus: Reliability, partial failure, and why remote calls cannot be treated like local calls.

Key Takeaways

• A network boundary turns direct-looking work into message exchange with incomplete evidence.
• A timeout proves the caller stopped waiting; it does not prove the receiver stopped working.
• Latency has a tail, and tail requests can cross caller timeouts even when averages look healthy.
• Partial failure means some participants keep working while others lack the evidence they need.
• Good boundary design uses deadlines, unknown states, owner checks, and signals tied to the real request path.