Observability for Worker Systems

By the end of this lesson, you will be able to...

• Distinguish a live worker fleet from a worker system that is making useful progress.

• Use queue age, depth, rates, retries, and in-flight work to form a failure hypothesis.

• Choose an alert and a next diagnostic step for delayed background work.

Idea in one sentence: A worker system is healthy when the right jobs finish within their delay budget, not merely when worker processes are running.

Core Insight

Suppose a learning platform is meant to send reminder emails for a live class at 10:00. At 10:15, support reports that students have not received them. The worker dashboard is reassuring: every worker has a green heartbeat, CPU is moderate, and no pod has restarted.

That dashboard answers a small question: can the processes still speak? It does not answer the product question: are reminders reaching people on time?

Background work can fail quietly. Jobs may wait in a queue, run slowly, retry a rejected provider request, or be crowded out by a bulk task. A worker can be alive through all of those states. Observability makes that hidden waiting visible, so an operator can tell whether to wait, add capacity, reduce admission, investigate a dependency, or protect one class of work from another.

The trade-off is deliberate: richer lifecycle evidence costs instrumentation and alert-maintenance work, but it replaces a comforting green process check with evidence about the outcome. The previous lesson added limits and backpressure so a worker pool does not overwhelm a dependency. This lesson supplies the evidence that says whether those controls are helping. The next lesson will move to balancing requests across service replicas; here, keep the focus inside the queued-work path.

The Small Incident

The reminder job has this path:

course starts soon
    -> reminder job is enqueued
    -> worker claims it
    -> email provider accepts it
    -> job is recorded as completed

The product promise is not “a worker starts eventually.” It is “a student receives a reminder before class begins.” Suppose that promise allows at most ten minutes of delay after the scheduled send time. That ten-minute limit is a delay budget. It gives the dashboard a meaning beyond “higher” and “lower.”

The naive dashboard watches only two things:

worker_heartbeat = green
queue_depth = 420

Neither tells us enough. A depth of 420 may be a harmless burst that drains in seconds. It may also contain one reminder that has waited eighteen minutes. A heartbeat may be green while every email call is rejected.

What we need to observe is the job's movement through time.

From “Alive” to “Making Progress”

Plain meaning:

A worker is alive when its process can still report in. The system is making progress when accepted work moves toward a useful outcome fast enough.

In this scenario:

All eight reminder workers emit heartbeats, but the oldest reminder has waited eighteen minutes and completions have fallen close to zero.

Technical name:

This distinction is liveness versus progress. Liveness is one health signal. Progress is a collection of flow and outcome signals.

Make the lifecycle explicit before choosing metrics:

enqueued -> waiting -> claimed -> running -> completed
                              |              |
                              |              +-> failed
                              |                    |
                              |                    +-> retry scheduled -> waiting
                              |
                              +-> timed out or dead-lettered

Each transition can leave evidence. A job that stays in waiting points toward admission or claiming. A job that stays in running points toward execution or a dependency. A job that cycles from running to retry scheduled points toward a repeated failure. A job in the dead-letter queue needs an operator decision rather than another automatic attempt.

This does not require tracking every job as a metric. It requires choosing aggregate signals that reveal where the flow changes shape.

The Signals That Describe Flow

Use a small, stable set of dimensions such as queue, job_type, priority, and outcome. They let an operator compare reminders with bulk exports without turning every job into a separate time series.

Metrics answer “how much, how long, and how often?” Logs and traces answer “which dependency call, payload, or code path?” They work together. A chart may show retries rising in the email-reminders queue; sampled traces can then show that the provider is returning 429 responses.

Keep unique values such as job_id, user_id, and an email address out of metric labels. They create a large number of time series that are expensive to aggregate and hard to query. Put those identifiers in structured logs or traces, then link from an aggregate signal to representative examples.

Check: A nightly import creates 10,000 jobs. Depth jumps from 50 to 10,000, but the oldest age stays below 40 seconds and completion rate matches enqueue rate. Is the queue unhealthy?

Think first, then reveal.

Answer: Not from this evidence. The queue is acting as a buffer for a burst, and the flow is keeping up. Depth is a volume signal, not a verdict. Keep watching age and the difference between arrival and completion before treating the burst as an incident.

A Worked Dashboard Investigation

Return to the late reminders at 10:15. The on-call engineer groups metrics by queue=email-reminders and compares the recent interval with a normal one.

The path is now inspectable:

input: normal reminder demand
  -> transition: workers claim jobs and call the provider
  -> intermediate state: provider returns 429; jobs are scheduled for retry
  -> output: only 2 successful completions/minute; oldest age reaches 18 minutes
  -> decision: protect the provider and prioritize reminders, rather than add more workers

The naive response would be to raise concurrency because the queue is growing. That would create more provider calls, more 429 responses, and more retries. The evidence instead supports the controls from Lesson 007: lower the send rate to the provider limit, apply bounded backoff, and reserve capacity for the time-sensitive reminder queue. Communicate the missed delay budget, because “healthy pods” is not the user outcome.

After mitigation, the dashboard should show a sequence, not an instant miracle: retry attempts fall, completion becomes greater than enqueue rate, depth shrinks, and finally oldest age returns below the ten-minute budget. That order proves the system is catching up.

Read Failure Shapes, Not One Number

Several failures can create a growing queue. The surrounding signals separate them.

| Failure shape | Signal pattern | Next check | Likely response | | --- | --- | --- | | Harmless burst | Depth rises; age stays low; completion matches arrivals | Is the delay budget intact? | Observe; do not scale on depth alone. | | Slow dependency | In-flight and duration rise; completions fall | Which dependency span became slow? | Bound concurrency, use timeouts, investigate dependency. | | Provider rejection | Retry rate rises with a repeated error code | Are 429 or 5xx responses concentrated in one provider? | Pace requests and use bounded backoff. | | Consumer stall | Age and depth rise; in-flight is near zero despite live workers | Can workers claim jobs? Are leases or broker connections failing? | Restore claiming before adding workers. | | Queue starvation | One priority class ages while another consumes slots | Which queues occupy the pool? | Reserve capacity or separate pools. |

This table is a hypothesis tool, not an automatic diagnosis. A retry spike can come from a slow database, a bad payload release, or a provider. Use metrics to choose the next piece of evidence; use logs and traces to test the cause.

Check: Every worker heartbeat is green. email-reminders has an oldest age of 14 minutes, in-flight work is zero, and workers log repeated broker authentication failures when trying to claim. What should the alert call this?

Think first, then reveal.

Answer: A progress failure or consumer stall, not a healthy fleet. The heartbeat shows that processes exist; failed claims show they cannot move queued work. The immediate investigation is broker access and worker credentials, while the age alert captures the broken product promise.

Alerts Need a Product Boundary

An alert should name an action and a reason. “Queue depth above 500” is often weak because a safe depth depends on job cost and normal burst size. “Oldest scheduled reminder is older than ten minutes for five minutes, while completion is below enqueue rate” is stronger. It ties a user-facing budget to evidence that the system is not recovering on its own.

One practical alert set for this service is:

• page when oldest reminder age exceeds the delivery budget and completion remains below enqueue rate;
• create an urgent ticket when retry ratio rises for one provider or job type;
• alert when dead-letter growth is non-zero for a critical workflow;
• alert separately when workers cannot claim work, even if their heartbeats remain green.

Alerts need ownership and a runbook. A useful runbook starts with the same sequence used above: identify the affected queue and age, compare arrival with completion, inspect in-flight and retries, then sample traces or logs for the failing transition. It also says which controls are safe to change. Raising concurrency is not a default repair when the dependency is already rejecting calls.

Trade-offs and Limits

Detailed worker telemetry improves diagnosis, but it costs instrumentation work, storage, dashboard design, and alert maintenance. Histograms and traces can be especially valuable for long-running jobs, but sampling may hide a rare payload failure. Tune them around the workflows whose delay matters.

Queue age also has limits. It reveals delayed work, but not whether a completed email was actually opened, nor whether a provider accepted a request but later failed delivery. Add an outcome signal at the boundary you control, and state clearly where responsibility ends.

Finally, observability does not repair a broken worker system. It reduces time spent guessing. The boundary becomes visible when an alert detects a delay or bad transition but the runbook still cannot identify an owner, dependency, or safe mitigation. That is a signal to improve the job lifecycle and its evidence, not merely to add more graphs.

Common Confusions

Confusion: A green worker means a healthy worker system

Why it is tempting:

Process health is simple and visible. It is also necessary: a crashed worker cannot finish jobs.

Better model:

Liveness is a prerequisite, not a result. Combine it with age, completion, retries, and job outcomes to see whether the workflow advances.

Confusion: A large queue always means more workers

Why it is tempting:

Depth looks like a direct measure of insufficient capacity.

Better model:

Compare depth with age, rates, in-flight work, and dependency errors. More workers help only when safe capacity is actually available. They worsen a throttled dependency.

Confusion: More metric labels always produce better visibility

Why it is tempting:

It feels useful to put every identifier on every graph.

Better model:

Use low-cardinality dimensions for aggregate behavior. Use logs and traces for one job, user, or payload. This preserves both dashboards and investigation detail.

Practice: Write the First Dashboard Note

A thumbnail-generation queue has these values for ten minutes: depth rises from 200 to 1,800; oldest age rises from 30 seconds to 12 minutes; enqueue rate is steady at 120/minute; completion falls from 120 to 25/minute; in-flight work remains at its configured maximum of 16; median execution duration triples; retry rate is unchanged.

Write a three-line dashboard annotation that states: the user impact, the most likely failure shape, and the next evidence to inspect. Do not prescribe more workers yet.

Model answer: “Thumbnail jobs are exceeding the twelve-minute delay budget while input remains steady and useful completion has collapsed. Full in-flight capacity plus a tripled execution duration suggests a slow handler or downstream dependency rather than a claim stall or retry storm. Inspect traces by dependency and handler stage, then apply a safe concurrency or timeout limit if that dependency is saturated.”

Resources

• [DOC] OpenTelemetry metrics — Focus: Counters, histograms, and stable attribute sets for describing job flow.
• [DOC] Prometheus instrumentation best practices — Focus: Choosing useful metrics and avoiding high-cardinality labels.
• [DOC] Celery monitoring and management — Focus: Events and task states exposed by a mature worker framework.
• [BOOK] Monitoring Distributed Systems — Focus: Designing alerts around user-visible symptoms and actionable signals.

Key Takeaways

• A live worker process is not proof of useful progress; the product outcome depends on jobs moving through their lifecycle within a delay budget.
• Queue age, arrival and completion rates, in-flight work, duration, retries, and dead letters describe different failure shapes together.
• Use aggregate metrics to find the broken transition, then logs and traces to test the cause and choose a safe response.
• An alert is strongest when it connects a user-facing delay budget to a concrete next action.