Observability and Debugging Distributed Systems
LESSON
Observability and Debugging Distributed Systems
By the end of this lesson, you will be able to...
trace one user-visible promise across requests, queues, workers, and durable records.
separate logs, traces, metrics, and business state by the questions each one answers.
design the minimum evidence needed to debug retries, partial side effects, and safe repair.
Idea in one sentence: Observability is the ability to reconstruct what happened to a user promise from joined evidence across system boundaries.
Core Insight
Maya uploads a contract PDF to a collaboration app.
The app says:
Upload complete.
Shared with Legal.
Five minutes later, Legal opens the link and sees:
File not available.
The dashboard says the upload API is healthy. Object storage shows normal latency. The preview worker has no obvious error spike. Each local statement may be true. None of them answers Maya's question:
Did my contract upload succeed?
If not, where did it stop?
Can the system repair it safely?
Distributed systems divide work. They also divide evidence.
One service sees the upload request. Object storage sees bytes written. A metadata database sees a file record. A queue sees a scan job. A worker sees virus-scan status. A sharing service sees permissions. A preview service sees whether the PDF can be rendered.
If those facts cannot be joined, debugging becomes a pile of plausible stories.
Plain meaning:
Observability means the system keeps enough connected evidence to reconstruct an important workflow after the fact.
In this scenario:
An engineer should be able to start from Maya's report and find the upload attempt, stored object, metadata record, scan job, sharing decision, preview job, and repair path.
Technical name:
This is observability for a distributed workflow. It is not just logs, metrics, or traces. It is the ability to answer a question about a user-visible promise.
The Naive Idea: Check Whether Services Are Healthy
The first instinct is to open dashboards.
upload API: 99.9% success
object storage: normal p95 latency
scan workers: mostly healthy
preview workers: low CPU
sharing service: no error spike
This is useful, but incomplete. Local health does not prove the workflow succeeded.
The failed upload may be one rare path:
bytes stored
metadata insert succeeded
scan message delayed
share link created before scan passed
preview worker could not find the file version
No single component owns the whole story. The user promise crosses boundaries. Observability has to preserve the evidence at those boundaries.
Check: If the upload API returned 200 OK, does that prove Legal can open the file?
Think first, then reveal.
Answer: No. It proves only that one request path returned success. Legal also needs durable bytes, a metadata record, scan status, sharing permission, and whatever availability rule the product promises before a link is usable.
Start With The Promise And The Identities
The product promise is not "one HTTP request returned success." A better promise is:
After the app says "Upload complete and shared,"
the intended recipients can open the exact uploaded file,
or the system can explain and repair the failed step.
That promise needs several identifiers.
file_id: file-913 durable business object
upload_session_id: upload-77 one intended upload
object_key: obj/contracts/913 stored bytes
request_id: req-A one HTTP attempt
trace_id: trace-44 one observed execution path
scan_message_id: msg-scan-301 queued scan attempt
preview_job_id: job-prev-82 preview-generation attempt
share_id: share-legal-12 permission decision
These ids are related, not interchangeable.
Maya may retry after a timeout. The retry should get a new request id because it is a new network attempt. It should keep the same upload session if it represents the same intended file upload. The scan queue may redeliver a message. That should not create a second file. The preview worker may run twice. That should not make two different official file records.
Stable identities let separate local facts join into one investigation.
The Evidence Has Different Jobs
Different observability signals answer different questions.
Structured logs record local facts at important boundaries:
req-A received
object write started
metadata insert committed
scan message published
share link created
preview job failed
Good logs include the ids needed to join the story. They do not need to log the full PDF contents.
Traces show timing and waiting across calls. A trace can reveal that upload spent 120 ms in the API, 900 ms writing bytes, 40 ms inserting metadata, and 3 seconds waiting to publish a queue message. Traces explain path and latency. They do not replace durable business records.
Metrics show patterns across many workflows. Scan queue depth, oldest scan age, preview failure rate, object-store write latency, and share-link open failures tell whether Maya's case is isolated or part of a broad condition. Metrics detect shape. They rarely explain one file alone.
Durable records and events answer state questions:
file record exists?
object checksum matches?
scan passed?
share permission committed?
preview generated?
repair action recorded?
For critical side effects, durable records are the stronger evidence. A trace span saying "metadata insert" is not the same as the committed metadata row.
The goal is not maximum telemetry. The goal is a joinable evidence set:
logs: local boundary facts
traces: path and timing
metrics: systemic pressure
records: durable workflow state
A Worked Trace: The Missing File Link
Follow Maya's upload.
1. The Upload Request Starts
09:15:00.000
request:
request_id=req-A
trace_id=trace-44
upload_session_id=upload-77
file_id=file-913
The upload API receives bytes and writes them to object storage.
09:15:00.840
object record:
object_key=obj/contracts/913
checksum=abc123
size=4.8 MB
status=stored
This proves bytes were stored. It does not prove the file is safe to share, indexed, previewable, or visible to Legal.
2. Metadata And Scan Work Are Created
The API commits a file record.
09:15:00.910
file metadata:
file_id=file-913
object_key=obj/contracts/913
owner=maya
state=awaiting_scan
Then it publishes a scan message.
09:15:00.980
queue:
scan_message_id=msg-scan-301
file_id=file-913
object_key=obj/contracts/913
trace_context=trace-44
The queue boundary matters. If the metadata exists but no scan message exists, the system is stuck before scanning. If the message exists but is old, the scan worker may be delayed. If the worker consumed it and failed, the file needs a different repair.
3. The App Creates The Share Too Early
The sharing service creates the Legal permission before the scan result arrives.
09:15:01.100
share record:
share_id=share-legal-12
file_id=file-913
recipient=legal-team
state=created
The UI says "shared." But the file state is still awaiting_scan.
This may be a product bug, not only an infrastructure bug. If the product promise says recipients can open the file after "shared," then the response text was too strong. It should have said "upload received, scanning" or waited for the scan state.
4. The Scan Worker Fails Without A Durable Result
The scan worker consumes the message.
09:15:05.300
worker log:
scan_message_id=msg-scan-301
file_id=file-913
result=object_fetch_timeout
But it crashes before writing a durable scan result or requeue decision.
Now the trace may show a failed worker span. That is useful. The durable state is more important:
file metadata:
state=awaiting_scan
last_scan_result=none
An engineer can distinguish:
not scanned yet
scanned and failed
scanned and clean
scan result missing after worker attempt
Those states lead to different repair actions.
5. Legal Opens The Link
09:20:00
Legal -> file service:
share_id=share-legal-12
file_id=file-913
The file service sees:
share exists: yes
metadata exists: yes
object exists: yes
scan passed: no evidence
preview exists: no
A safe response might be:
File is still processing.
or:
File unavailable because scan did not complete.
The unsafe response is a vague "file not found" when the real state is "bytes exist, scan result missing." That hides the repair path.
So far, observability has turned one complaint into a precise state:
bytes stored
metadata committed
share created too early
scan result missing after worker attempt
recipient sees unavailable file
Build A Timeline, Then Mark Facts And Inferences
An incident timeline should separate what is known from what is guessed.
09:15:00.000 fact: req-A started upload-77 for file-913
09:15:00.840 fact: object obj/contracts/913 stored with checksum abc123
09:15:00.910 fact: metadata file-913 committed as awaiting_scan
09:15:00.980 fact: scan message msg-scan-301 published
09:15:01.100 fact: share share-legal-12 created
09:15:05.300 fact: worker logged object_fetch_timeout
09:20:00.000 fact: Legal opened share and saw unavailable
"Object storage caused the problem" is an inference until storage logs, worker retries, and object availability support it. "The scan never ran" is false if the worker log proves one attempt ran. "The user uploaded the wrong file" is an inference unless checksum, size, or client telemetry supports it.
This discipline matters because telemetry is imperfect. Clocks drift. Logs may be sampled. Workers can crash before writing final state. Messages can be delivered more than once. Good observability says which facts are durable, which facts are local, and which facts are missing.
Check: Which evidence is stronger for "the file can be shared": a trace span named create_share or a durable file record with scan_status=clean and a committed share record?
Think first, then reveal.
Answer: The durable records are stronger. The trace helps explain the path and timing, but the durable records prove the state that later services should rely on.
Instrument Boundaries, Not Just Services
Distributed failures often hide at boundaries.
For the upload workflow, important boundaries include:
client -> upload API
upload API -> object storage
upload API -> metadata database
metadata database -> scan queue
scan queue -> scan worker
scan worker -> file state
file state -> sharing service
sharing service -> recipient open
At each boundary, preserve enough context to answer:
what logical operation is this?
which business object does it affect?
what outcome occurred?
what evidence proves the next handoff or state transition?
For a queue, record enqueue time, message id, business id, delivery attempts, and final worker decision. For a retry, keep the same upload session or idempotency id if it is the same intended file. For repair, record why the repair ran, what evidence it used, and what state it changed.
Absence must be queryable. If a file has metadata but no scan result after ten minutes, the system should let an engineer find that gap directly. Missing evidence is often the clue.
Trade-offs And Limits
The trade-off is evidence versus cost.
Useful observability costs storage, CPU, bandwidth, indexing, attention, and privacy budget. Logging full documents or personal data is dangerous. High-cardinality labels can overload a metrics system. Sampling may hide a rare trace that matters. Retaining every payload forever creates operational and legal risk.
Good observability is designed around recovery. Keep the smallest set of facts needed to:
- detect that a user promise broke;
- connect local evidence across boundaries;
- identify durable side effects;
- choose a safe repair action;
- explain the outcome later.
It can still fail when ids are inconsistent, events are emitted before commits, trace context is dropped at queues, logs omit business ids, or repair jobs do not record their decisions.
Rare paths deserve deliberate retention rules, because routine sampling can hide exactly the evidence needed for repair.
Useful signals include:
- uploads with metadata but no stored object;
- uploads with object and metadata but no scan result after a deadline;
- scan queue depth, oldest message age, retry count, and dead-letter count;
- share links created before scan completion;
- recipient open failures grouped by file state;
- trace-context propagation failures across queues; and
- sensitive-field redaction and telemetry retention health.
If dashboards say every service is healthy but these workflow signals are bad, the user promise is still broken.
Common Confusions
Confusion: Logs equal observability
Why it is tempting:
Logs feel like the raw truth of what happened.
Better model:
Logs are local evidence. Observability requires joinable logs, traces, metrics, and durable records that answer a workflow question.
Confusion: A trace proves the business state
Why it is tempting:
A trace looks like the whole path.
Better model:
A trace shows observed execution and timing. Durable records prove whether important state transitions actually committed.
Confusion: More telemetry is always better
Why it is tempting:
When debugging hurts, collecting everything feels safe.
Better model:
Telemetry has cost and privacy risk. Keep evidence that supports detection, diagnosis, and safe repair.
Practice
Choose one workflow:
file upload
password reset
message send
seat booking
account deletion
webhook delivery
Fill in:
user-visible promise:
durable business id:
operation or idempotency id:
per-attempt request id:
trace context boundaries:
durable state transitions:
queue or worker evidence:
metric that detects a systemic version:
missing fact that must be queryable:
safe repair action if the workflow stops midway:
Model answer for message send:
user-visible promise:
after "sent", the message is durably stored or the sender can see a clear failure
durable business id:
message_id=msg-44
operation id:
send_operation_id=send-abc
request id:
one id per HTTP attempt
trace boundaries:
API request, storage write, fanout queue publish, delivery worker
durable transitions:
accepted, stored, queued_for_delivery, delivered_or_failed
queue evidence:
fanout message id, enqueue time, attempts, final result
systemic metric:
oldest undelivered fanout message age
missing fact:
messages stored but never queued for delivery
safe repair:
republish delivery job using message_id, not create a new message
Resources
- [REFERENCE] OpenTelemetry: Signals - Focus: How traces, metrics, logs, and related signals complement each other.
- [PAPER] Dapper, a Large-Scale Distributed Systems Tracing Infrastructure - Focus: Why distributed tracing exists and how request paths are reconstructed across boundaries.
- [BOOK] Site Reliability Engineering: Monitoring Distributed Systems - Focus: Monitoring signals, alerting, and the limits of aggregate summaries.
Key Takeaways
- Observability reconstructs a user-visible promise from joined evidence across service boundaries.
- Request ids, trace ids, operation ids, object ids, queue ids, and business ids answer different questions.
- Logs, traces, metrics, and durable records have distinct jobs; none replaces the others.
- The best telemetry set is the smallest one that detects broken promises, identifies side effects, and supports safe repair.