Observability Across Network Boundaries

By the end of this lesson, you will be able to...

• Reconstruct one cross-service request from metrics, logs, traces, and network evidence.

• Decide which signal answers which incident question.

• Spot missing telemetry for retries, routing, stale discovery, deadlines, and degraded success.

Idea in one sentence: Observability is the decision trail that lets you explain what each boundary saw, chose, retried, skipped, or could not know.

Core Insight

Imagine a learner clicks Complete lesson and waits ten seconds.

The browser shows a spinner. Then it shows success.

The frontend log says:

POST /complete-lesson took 9800 ms
final status = 200

The gateway log says:

first attempt timed out after 2500 ms
retry attempt succeeded

The progress service log says:

write committed normally
idempotency key = req-3018-a

The database metric says:

write queue p95 briefly rose to 1800 ms

The service-discovery event says:

endpoint P3 was draining during the first attempt

No single component thinks it has the whole story. The browser saw a slow success. The gateway saw a timeout and retry. The progress service saw a normal commit. The database saw a short queue spike. Discovery saw a route to an endpoint that was leaving rotation.

This is the observability problem created by network boundaries.

Once a request crosses a browser, gateway, proxy, service, database, queue, and background worker, each participant sees only its local slice. A caller timeout does not prove that the server failed. A server commit does not prove that the caller received the answer. A retry may hide the first symptom. A successful final response may hide stale data, skipped work, or burned retry budget.

Good observability is not "more logs." It is the ability to reconstruct the path:

what was requested
which boundary handled it
which policy decision happened
which endpoint was chosen
which retry or fallback occurred
which deadline remained
which result became visible to the user

The useful design shift is to instrument the path, not just the components. A dashboard per service is useful, but a networked incident often lives between services, in the decisions made at boundaries.

The Naive Debugging Model

The naive model says:

If the user saw a slow request, find the slow service.

Sometimes that works. Often it is too simple.

For a networked request, "slow" may mean several different things:

• the browser waited for the gateway
• the gateway waited for a retry deadline
• the first upstream endpoint was draining
• the second upstream endpoint deduplicated an earlier write
• the database queue was briefly saturated
• an optional dependency was skipped to preserve the final deadline
• DNS, TLS, TCP, or packet loss delayed connection setup

Those facts live in different places. If each place records only its own local outcome, the incident looks contradictory.

Plain meaning:

Observability is how you keep enough breadcrumbs to tell one request story after the request has crossed many boundaries.

In this scenario:

The request was slow because a first attempt hit a bad path, a retry succeeded, and the final success hid the cost.

Technical name:

This is cross-boundary observability: metrics, logs, traces, events, and sometimes packet-level evidence connected by shared context.

Shared Context Makes The Path Reconstructable

The first requirement is a durable request identity.

That identity may be a request ID, trace ID, correlation ID, idempotency key, or a combination. The exact name matters less than the rule:

The identity must survive boundaries.

For the learning platform, the request might travel like this:

browser
  -> frontend server
      -> API gateway
          -> progress-service
              -> progress database
          -> notification queue
              -> notification worker

The trace or request context should survive:

• the browser request
• gateway routing
• service-to-service calls
• retries
• database work
• queue publication
• async worker processing
• fallback or degradation decisions

If the progress service emits a completion event, but the notification worker starts a brand-new trace with no link to the original request, the story breaks exactly where delayed work begins.

Useful context does not mean "put every value everywhere." Metrics should keep labels bounded. Logs and traces can carry unique request details. The trick is to put facts in the place that can carry them safely.

For one completion request, useful fields include:

trace_id = tr-3018-a
request_id = req-3018-a
operation_class = authoritative_write
idempotency_key = lesson-3018-user-7
caller_deadline_ms = 10000
gateway_attempt = 1 or 2
selected_endpoint = P3 or P1
endpoint_version = v1 or v2
route_reason = write_ready, retry_after_timeout, or draining_rejected
deduplication_result = first_commit or duplicate_suppressed
final_user_status = 200
degraded_facts = notification_deferred

With those fields, separate local observations can become one incident timeline.

Metrics, Logs, Traces, Events, And Packets

Different signals answer different questions.

Metrics answer aggregate questions:

Is this getting worse?
How many users are affected?
Which dependency is saturated?
Did retries increase after the deploy?
Is one zone slower than the others?

Examples:

progress_completion_latency_p95 = 9.8 s
gateway_retry_rate = 14%
db_write_queue_depth = 120
zone_c_connection_reset_rate = elevated

Logs answer discrete fact questions:

What did this component decide?
Which idempotency key was used?
Why was a retry allowed?
Why was a fallback chosen?

Examples:

request=req-3018-a attempt=1 route=P3 result=timeout deadline_remaining=7400ms
request=req-3018-a attempt=2 route=P1 result=success dedupe=duplicate_suppressed

Traces answer shape and timing questions:

Which hops participated?
Where was time spent?
Which span retried?
Which endpoint answered?
Which downstream work was skipped?

A useful trace can show:

browser span: 9800 ms
gateway span: retry after 2500 ms
progress-service attempt 1: endpoint P3, timeout
progress-service attempt 2: endpoint P1, deduplicated success
notification span: skipped, deadline exhausted

Events answer state-change questions:

When did P3 start draining?
When did discovery remove P3?
When did the deploy start?
When did the circuit breaker open?

Packet captures and lower-level network tools answer movement questions below the application:

Was DNS slow?
Did TLS negotiation fail?
Were packets retransmitted?
Did connections reset?
Was traffic unexpectedly crossing zones?

The trade-off is abstraction versus evidence. Application telemetry explains meaning. Network telemetry explains movement. A strong investigation can use both without mixing them into one vague phrase like "the network was slow."

Check: A trace shows the progress-service span waited 2.5 seconds, but it does not show DNS timing or TCP retransmissions. Can you conclude packet loss was not involved?

Think first, then reveal.

Answer: No. An application trace can show where application time was spent, but lower-level network behavior may require resolver logs, proxy stats, connection metrics, or packet evidence.

A Worked Incident Trace

Now reconstruct the slow completion request.

Input:
  learner clicks Complete lesson
  request_id = req-3018-a
  operation = POST /complete-lesson
  operation_class = authoritative_write
  deadline = 10 seconds

The transition across boundaries looks like this:

t0 browser sends request with trace_id=tr-3018-a
t1 gateway resolves progress-service
t2 discovery cache still includes P3
t3 gateway routes attempt 1 to P3
t4 P3 is draining and slow to reject new work
t5 gateway times out attempt 1 after 2500 ms
t6 gateway retries because idempotency key exists and deadline remains
t7 discovery refresh removes P3
t8 gateway routes attempt 2 to P1
t9 P1 sees the same idempotency key and commits or deduplicates safely
t10 notification work is deferred because little deadline remains
t11 browser receives final 200 after 9800 ms

The intermediate state is not one clean success or failure:

The output or decision is:

The system returned success, but it was degraded success.
The write path survived because the retry had an idempotency key.
The route to P3 was the likely first failure point.
The notification side effect was deferred because the deadline was almost spent.

The naive failure contrast:

Naive investigation:
  "The progress service returned 200, so it was fine."

Better investigation:
  "The final response was 200, but the path used a retry, stale discovery,
   endpoint draining, deduplication, and deferred async work."

So far:

• the user outcome was slow success
• the path included a failed first attempt
• the retry was safe only because the operation had idempotency
• stale discovery and draining were part of the route decision
• final success did not mean the system was healthy

Tail Latency Hides In Fan-Out

Many user requests call several dependencies.

The lesson page may need:

catalog metadata
progress summary
recommendations
certificate eligibility
notification preferences

Even if each dependency is usually fast, the page is exposed to the slow tail of the required work. A request that needs four dependencies has more chances to hit one slow dependency than a request that needs one.

Example:

page request deadline = 1000 ms

catalog:         80 ms
progress:       120 ms
recommendation: 760 ms
certificate:    skipped after deadline budget is almost gone

If the final page returns 200 OK, the system can look healthy. But internally it spent most of the deadline on one optional dependency and skipped another. That is important even when the user did not see an error.

Observability should therefore capture budgets and dependency contribution:

total deadline
time spent per dependency
retries per dependency
fallback chosen
optional work skipped
stale data used
remaining deadline at each boundary

Without those facts, a team may optimize the wrong service. The problem may not be the slowest average dependency. It may be the dependency that sometimes consumes the whole tail.

Degraded Success Is Still A Signal

Some network failures are hidden by good product behavior.

The page loads, but recommendations are omitted. The catalog appears, but from a stale cache. The progress write succeeds, but only after a retry. The certificate job is deferred because the system cannot confirm authoritative progress state before the deadline.

That may be the right user experience. But it is still a signal.

final status: 200 OK
degraded facts:
  recommendations_skipped=true
  catalog_cache_age=45s
  progress_retry_count=1
  certificate_check_deferred=true

If observability records only final failures, all of those facts disappear. The system looks healthy while it is spending retry budget, hiding optional dependency failures, or serving older data than expected.

The trade-off is signal versus noise. Not every fallback deserves an alert. But degraded success should be measurable, searchable, and visible in dashboards or traces. Otherwise the first visible outage arrives after the resilience mechanisms have been quietly overloaded for hours.

Check: The completion page returns 200 OK, but the trace shows one retry and notification_deferred=true. Should this count as completely healthy traffic?

Think first, then reveal.

Answer: No. The user-visible result succeeded, but the path consumed resilience mechanisms. It should be recorded as degraded success, even if it does not page anyone.

Design The Telemetry Around Decisions

A useful observability design asks:

What decisions can change the request path?
What evidence will prove each decision happened?

For this track, important decisions include:

This connects directly to earlier lessons. Timeouts and retries are policy decisions. Health checks and load balancing are routing decisions. Discovery and identity are control-plane decisions. Partitions create uneven knowledge. Observability should show those decisions, not only their symptoms.

Common design mistake:

Record "upstream failed."

Better:

upstream=progress-service
endpoint=P3
attempt=1
route_reason=cached_discovery_candidate
result=timeout
deadline_remaining_ms=7400
next_action=retry_with_idempotency_key

The better version is longer, but it is not just more verbose. It preserves the decision trail.

Trade-offs And Limits

More telemetry is not automatically better.

High-cardinality metric labels can make a metrics system expensive or unusable. User IDs, request IDs, raw URLs, and idempotency keys usually do not belong as metric labels. They are better suited for logs, traces, or sampled events.

Tracing every span at full detail can cost too much. Sampling helps, but sampling can hide rare incidents unless errors, retries, slow paths, and degraded success are kept at higher sampling priority.

Logs can preserve rich facts, but unstructured logs become hard to query. Structured logs with consistent names are easier to join with traces and metrics.

Packet captures are powerful, but they are detailed, sensitive, and often hard to keep continuously. Use them when the question is below the application layer.

The boundary is visible when engineers cannot answer:

Which request path was affected?
Which policy decision changed the path?
Which layer observed the failure?
Which layer guessed?
Which evidence is missing?

Good observability does not remove failure. It reduces the time spent arguing about which local truth matters.

Common Confusions

Confusion: A 200 Means The Request Was Healthy

Why it is tempting:

The user got a successful response, so the incident seems over.

Better model:

A final 200 OK can hide retries, stale reads, skipped optional work, deferred side effects, or a near-deadline path. Record degraded success separately from clean success.

Confusion: Traces Replace Metrics

Why it is tempting:

Traces feel detailed because they show one path.

Better model:

Traces explain shape. Metrics explain scale. You usually need metrics to see that a problem is spreading and traces to understand one example path.

Confusion: The Network Is Slow

Why it is tempting:

Many symptoms cross the network, so the network becomes the blame bucket.

Better model:

Separate application waiting, routing decisions, discovery staleness, dependency saturation, DNS/TLS/connection behavior, and packet movement. Each needs different evidence.

Practice

Review this incident report:

Symptom:
  3% of completion requests took more than 8 seconds.

Known facts:
  final status was usually 200
  gateway retry rate rose from 1% to 12%
  one zone had a deploy draining progress-service P3
  database write queue p95 rose briefly
  notification jobs fell behind by 4 minutes

Write a five-line observability plan:

metric to check:
trace field to require:
log field to require:
network or control-plane event to inspect:
degraded success fact to record:

Model answer:

metric to check: gateway retry rate by operation class and zone
trace field to require: selected endpoint, attempt number, deadline remaining
log field to require: idempotency key and deduplication result
network or control-plane event to inspect: P3 draining and discovery cache age
degraded success fact to record: notification_deferred and retry_count

The exact names can differ. The important habit is to connect each question to a signal.

Resources

• [ARTICLE] OpenTelemetry Concepts - Compare traces, metrics, logs, context propagation, and semantic conventions.
• [BOOK] Site Reliability Engineering: Monitoring Distributed Systems - Use monitoring to detect symptoms and preserve service promises.
• [ARTICLE] Envoy Observability - Study what a network proxy can reveal about retries, routing, and upstream behavior.

Key Takeaways

• Networked incidents need path observability, not only per-service dashboards.
• Shared request context connects caller timeouts, server commits, retries, routing, and async work into one story.
• Metrics, logs, traces, events, and packet evidence answer different incident questions.
• Observability should record policy decisions such as retries, routing, discovery cache use, fallback, and deadline exhaustion.
• Degraded success is still a signal when final success hides retries, stale data, skipped dependencies, or deferred work.