@jepemo
EXIT

Observability Basics for Backend Services

Learning / Architecture And Platforms / Backend Development Foundations / 015

LESSON

Backend Development Foundations

015 25 min beginner

Observability Basics for Backend Services

By the end of this lesson, you will be able to...

  • Trace one failed backend request through logs, metrics, traces, and events.

  • Choose signals that connect service behavior to user-visible promises.

  • Explain why "many logs" can still leave a backend hard to operate.

Idea in one sentence: Observability is the evidence a backend emits so a team can explain what happened, where it happened, and what to do next without guessing from symptoms alone.

Core Insight

The previous lesson shipped a release carefully. The image was built, the migration was compatible, and rollout checks watched for harm.

Now imagine the release is live.

At 10:07, support receives this report:

I clicked Place order.
The page showed "Something went wrong."
My request id was req_982.

The orders API has many log lines. Some of them show stack traces. A dashboard also shows CPU, memory, and request count.

But the team still cannot answer the useful questions:

Did this request reach POST /orders?
Which release handled it?
Did auth pass?
Did validation pass?
Did the database commit?
Did payment time out?
Is this one user, one route, or everyone?

This is the beginner trap:

We have logs, so we have observability.

Logs are only one kind of evidence. They are useful when they contain the right fields and connect to the rest of the system.

Observability means designing the service to leave an evidence trail. That trail should follow a request across important boundaries:

client -> route -> auth -> validation -> database -> dependency -> response

The goal is not to record everything. The goal is to record the evidence needed to explain behavior, protect user promises, and make operational decisions.

The Production Symptom

Use one recurring scenario:

POST /orders sometimes returns 500 after release abc123.

The request path is familiar from earlier lessons:

client request
  -> route POST /orders
  -> authentication and authorization
  -> JSON validation
  -> database transaction
  -> payment provider call
  -> API response

The user sees only the end of the path:

status=500
request_id=req_982

The service should know more than that.

It should emit evidence at the boundaries where important things happen:

Plain meaning:

The service leaves enough clues to reconstruct what happened.

In this scenario:

A responder can start with req_982 and find the route, release, database state, payment result, and final response.

Technical name:

That designed evidence is observability.

What The System Knows

Observability usually uses four related tools.

Logs are records of discrete facts. They answer questions like:

What happened for this request?
Which error category did we return?
Which release wrote this line?

A useful log is structured:

level=error
event=route_failed
request_id=req_982
route=POST /orders
status=500
error_code=payment_timeout
release=abc123
duration_ms=3120

Metrics are numbers over time. They answer questions like:

Did checkout failures increase?
Is latency rising?
Is the payment provider timing out more often?

Useful beginner metrics include:

http_requests_total{route=POST /orders,status_class=5xx}
http_request_duration_ms{route=POST /orders}
payment_provider_timeouts_total{provider=payco}
orders_created_total

Traces connect work for one request. A trace is made of spans. Each span records a piece of work, such as auth, validation, database transaction, or payment authorization.

Structured events record meaningful facts. They are not random debug strings. Examples:

order_created
payment_authorization_failed
idempotency_replayed
job_retry_scheduled

These tools answer different questions.

A metric can say "checkout failures jumped at 10:05." A trace can say "this request spent three seconds in payment authorization." A log can say "request req_982 failed with payment_timeout on release abc123." An event can say "no order was created."

So far, observability is not a dashboard collection. It is an evidence path.

A Request Evidence Trace

Trace one healthy request first:

Input:
  POST /orders
  request_id=req_481
  release=abc123

Step 1: request starts
  log event=route_started route=POST /orders request_id=req_481 release=abc123
  metric http_requests_total{route=POST /orders,status=started} += 1

Intermediate state:
  the request is visible and correlated

Step 2: service checks the request
  span auth.check duration=8ms status=ok
  span validate_json duration=4ms status=ok

Step 3: service changes durable state
  span db.transaction duration=35ms status=committed
  event order_created order_id=ord_1001

Step 4: service calls dependency
  span payment.authorize duration=180ms status=ok

Output:
  response status=201 duration_ms=252
  log event=route_finished status=201 request_id=req_481 duration_ms=252
  metric http_request_duration_ms{route=POST /orders,status_class=2xx} records 252

Now trace the failure:

Input:
  POST /orders
  request_id=req_982
  release=abc123

Step 1: request starts
  log event=route_started route=POST /orders request_id=req_982 release=abc123

Step 2: local checks pass
  span auth.check duration=7ms status=ok
  span validate_json duration=5ms status=ok

Intermediate state:
  the request is valid enough to attempt payment

Step 3: dependency stalls
  span payment.authorize duration=3000ms status=timeout provider=payco
  event payment_authorization_failed reason=timeout

Output:
  response status=500 error_code=payment_timeout
  log event=route_failed request_id=req_982 status=500 error_code=payment_timeout
  metric checkout_failures_total{reason=payment_timeout,release=abc123} += 1

The naive failure contrast is:

bad evidence:
  "Exception: timeout"

usable evidence:
  req_982 -> POST /orders -> release abc123 -> payment.authorize timeout -> 500

With the usable evidence, support can connect the user report to one request. Engineering can see the failing dependency. The release field lets the team compare before and after abc123. Metrics show whether this is isolated or widespread.

Check: Why is request_id useful in logs and traces but usually dangerous as a metric label?

Think first, then reveal.

Answer: request_id identifies one request, so it helps connect detailed evidence. But metrics need bounded labels that aggregate many requests. A unique label for every request creates very high cardinality, which can make metrics expensive and hard to query.

Signals Start From Promises

Do not start observability design with "what can we collect?"

Start with "what promise are we trying to keep?"

For the orders API, useful promises are:

POST /orders creates one order or returns a clear recoverable error.
GET /orders/:id only returns orders the caller may see.
Payments should not time out often enough to break checkout.
Deployments should not sharply increase error rate.

Each promise suggests signals.

checkout promise:
  request count by route
  success rate
  4xx vs 5xx rate
  latency percentiles
  validation failure rate
  payment timeout rate
  idempotency conflict rate

authorization promise:
  deny count by route and action
  admin route access failures
  suspicious repeated denies

release promise:
  error rate by release
  latency by release
  container restart count
  readiness failure count
  migration version

This matters for alerts.

An alert should usually point at user harm or likely user harm:

checkout success rate dropped below threshold

That alert is stronger than:

one internal debug line appeared

Internal signals are still useful. CPU, memory, queue age, database connection pool pressure, and dependency error rate can explain why user harm is happening. But if every internal wiggle pages someone, responders stop trusting alerts.

Good observability separates:

symptom:
  users cannot place orders

possible causes:
  payment provider timeout
  database lock
  bad release
  validation bug
  exhausted connection pool

The page should make the symptom visible. The linked dashboard or runbook should help investigate likely causes.

What Can Still Go Wrong

Observability has its own failure modes.

Confusion: More logs means more observability

Why it is tempting:

Logs feel concrete. If the service prints a lot, it looks like there is a lot of evidence.

Better model:

Useful evidence is connected, structured, and safe. A thousand unstructured lines without request ID, route, release, or error category may be less useful than five well-designed records.

Confusion: Metrics should include every detail

Why it is tempting:

Labels make metrics easy to filter, so adding more labels feels helpful.

Better model:

Metrics should aggregate. Use bounded labels such as route template, status class, dependency name, and release. Avoid raw URL, email, order ID, request ID, token, or arbitrary exception message.

Confusion: Traces replace logs

Why it is tempting:

A trace is visual and often shows the request path clearly.

Better model:

Traces show where time and work went. Logs preserve specific facts and decisions. Metrics show trends. Events preserve domain facts. The tools cooperate.

Confusion: Observability is only for incidents

Why it is tempting:

People notice missing evidence when something breaks.

Better model:

Observability is part of design. It supports support tickets, release checks, capacity decisions, security review, debugging, and capstone readiness.

Security is a real boundary. Do not log raw bearer tokens, passwords, payment card data, secret keys, private request bodies, or sensitive personal data. A trace attribute can leak data just as easily as a log line. Redaction should be deliberate.

Check: An API logs every exception, but logs do not include route, request ID, release, or dependency name. What will be hard during an incident?

Think first, then reveal.

Answer: The team will struggle to connect a user report to one request, group failures by endpoint, compare failures before and after a release, and identify whether a dependency is involved. The logs contain errors, but not enough operating context.

Background Work Needs Evidence Too

Not every backend failure happens while a user waits for an HTTP response.

POST /orders may return 202 Accepted after enqueueing work:

authorize payment
send receipt email
notify warehouse

If a worker fails silently, the HTTP edge can look healthy while real business work piles up.

Worker evidence should include:

event=job_started
job=authorize_payment
job_id=job_77
request_id=req_982
attempt=1
queue=payments

event=payment_authorization_failed
job_id=job_77
order_id=ord_1001
reason=timeout

event=job_retry_scheduled
job_id=job_77
next_attempt_in_ms=30000

Useful worker metrics include:

queue_depth
oldest_job_age
job_success_count
job_retry_count
dead_letter_count
processing_latency

The same principle applies: instrument boundaries. A job boundary needs job ID, queue name, attempt number, duration, outcome, and safe error category. If the job came from a request, carry the request ID or trace ID when possible.

Trade-offs and Limits

Observability improves the ability to explain behavior. It does not make the system correct by itself.

The central trade-off is visibility versus cost, noise, and risk. More evidence can make incidents easier to investigate, but every signal has a price. Logs need storage. Metrics need careful labels. Traces add overhead and sampling decisions. Alerts need owners. Sensitive fields need redaction. A good backend does not collect everything it can see. It collects the evidence that helps someone make a safer decision.

This helps when:

It costs:

It does not protect us from:

You can see the boundary when:

The practical rule is:

collect evidence that supports a decision

If evidence does not help someone investigate, alert, debug, audit, support a user, or validate a release, it may be noise.

Practice

Design the smallest observability plan for this incident:

Symptom:
  After release abc123, POST /orders returns intermittent 500s.

Known path:
  validation -> auth -> database transaction -> payment provider -> response

Support has:
  request_id=req_982

Answer these:

  1. Name three log fields you need.
  2. Name two metrics that show whether this is widespread.
  3. Name one trace span that would localize latency or failure.
  4. Name one sensitive field that should not be logged.
  5. Name one alert tied to user-visible behavior.

A strong answer:

Connections

Lesson 014 showed that a release needs gates and signals. This lesson explains what those signals are made of.

Lesson 016 uses observability as part of the capstone review. A backend is not ready just because it responds once. It is ready when one important request can be explained from client contract to durable state to production evidence.

Resources

Key Takeaways

PREVIOUS CI/CD, Migrations, and Release Checks NEXT Backend Service Capstone: Build, Ship, and Operate One API

← Back to Backend Development Foundations

← Back to Architecture And Platforms

← Back to Learning Hub