Capstone: Diagnose a Noisy System

By the end of this lesson, you will be able to...

• Build an uncertainty model for a changing technical system.

• Combine distributions, sampling, comparison, measurement, time dependence, hidden state, and queue evidence.

• Communicate a decision with explicit support, uncertainty, and next measurements.

Idea in one sentence: A good diagnosis is not the most confident story; it is the smallest model that makes the evidence, alternatives, and remaining uncertainty inspectable.

Core Insight

Consider a document-indexing API after a deployment. The dashboard says the new version is faster: mean server latency fell from 118 ms to 110 ms. At the same time, customer support reports occasional ten-second waits, one region shows queue growth, and the tracing agent was upgraded in the same release.

There is no single number that resolves the incident. You need a model that can represent ordinary requests, rare tails, missing traces, bursts, local capacity, and a possibly degraded dependency. This capstone walks through that dossier and then leaves the final transfer to you.

The Scenario

The indexing service runs in two regions, East and West. Each region has two workers. A worker can process four indexing jobs per minute. The deployment changed application code and tracing configuration at the same time.

The team collects these summaries for 30-minute windows:

The after window also contains a scheduled customer import. The import sends repeated keys to West because of locality in the routing layer. The new tracing agent adds an estimated 12 ms to recorded server spans, but drops spans more often when workers are overloaded.

The decision is whether to continue the rollout, pause it, or roll back. A useful dossier must distinguish at least four hypotheses:

• H1: Process change. The deployment or import makes indexing genuinely slower.
• H2: Measurement change. The tracing agent changes the recorded span value or boundary.
• H3: Selection change. Slow or failed spans disappear from the after dashboard.
• H4: State and capacity change. A burst or hot routing key creates a degraded queue episode in West.

These hypotheses can all move a dashboard. The job is to collect evidence that separates them.

Constraints

The team cannot stop indexing globally. It can pause the import, route a small canary to the old version, compare client and load-balancer timing, and inspect queue counters. The incident window is short, so the first response must use signals already available.

The dossier must:

1. define the outcome and measurement boundary;
2. state the population and missingness rule;
3. represent ordinary variation and tail risk;
4. account for time, routing, and hidden operating state;
5. compare plausible explanations with an independent signal;
6. state what the evidence does not establish.

Design Goal

Produce a conclusion that is actionable but reversible:

Which change should the team make now, what evidence supports it, and which observation would change the decision?

Do not optimize for a single precise latency estimate. Optimize for a decision that remains safe if one assumption is wrong.

Proposed Model

Start with the model card:

• \(X_i\): true user-visible completion time for request \(i\), from client send to successful response or timeout.
• \(Y_i\): recorded server span.
• \(B\): systematic tracing overhead, estimated at 12 ms after the deployment.
• \(\epsilon_i\): random timing error.
• \(R_i\): indicator that a span is recorded.
• \(N_t\): indexing jobs arriving in minute \(t\).
• \(Q_j(t)\): unfinished jobs at worker \(j\) after service.
• \(S_t\): hidden operating state, such as healthy or dependency-degraded.

The observation path is:

The queue path is:

where \(A_j(t)\) is work routed to worker \(j\) and \(c_j=4\) jobs per minute. The state path is described by transition and observation probabilities: a degraded dependency can persist, and it makes long waits more likely without making every request fail.

This is enough structure for the decision. It is not a claim that the service has only two states or that every timing error is additive.

Walkthrough

1. Freeze the boundary and population

Use client-observed completion time as the primary outcome. Include successes, timeouts, and retries in the population; report retries separately so they are not mistaken for independent new work. Keep the same endpoint, region, tenant mix, and load range in both windows.

The server-span mean cannot be the primary decision metric because its overhead and missingness changed. It remains useful as a diagnostic after its observation model is recorded.

2. Separate ordinary variation from tail risk

The client median improved by 2 ms, but p95 worsened by 40 ms and timeout rate increased. The mean server span fell because the after dashboard omitted 8% of spans and added a fixed offset that does not explain the client result. The tail is the pressure that matters for user-visible reliability.

Compare the full client distribution, not only its mean:

The small median gain does not cancel the tail regression. A decision that values predictable completion should prioritize the p95 and timeout evidence.

3. Inspect sampling and missingness

Recorded spans fell from 99.2% to 92.0%. If overload makes a span more likely to be dropped, the after server metric is missing not at random. The observed 110 ms mean is a selected subset, not a faithful summary of all requests.

Compare expected request counts with recorded spans by region, status, latency bucket, and queue level. If missingness rises exactly when West queues grow, H3 gains support. A longer span sample would only make the selected subset more precise.

4. Test the measurement hypothesis

Compare client timing, load-balancer timing, and server spans for the same request identifiers. A fixed 12 ms agent overhead should produce a roughly stable offset on recorded spans, while dropped spans should affect counts and tail composition. If client and load-balancer timing agree on the tail increase but server spans do not, the process changed and the instrument is also distorting the magnitude.

The comparison should also check whether the new agent changed the start or stop boundary. A span that begins after queue admission cannot see time spent waiting in the worker queue.

5. Trace time, locality, and state

Plot one-minute arrivals and queue length by region. The scheduled import sends repeated keys to West, so \(A_{\text{West}}(t)\) may exceed its local capacity even when the global rate looks safe. The queue maximum of 41 jobs is evidence that the system has memory: work left by one minute delays later jobs.

Look for a hidden dependency state. If database-pool wait time, cache misses, or downstream latency rises before West’s queue grows, a degraded state may be causing both queueing and long requests. A single successful request after the peak does not prove recovery; inspect persistence and the recovery slope.

6. Run a controlled comparison

Pause the import for a short safe interval, keep a small old-version canary, and preserve the same client measurement. The prediction table is:

The table does not prove a cause by itself. It specifies what evidence would update each hypothesis.

7. Use a small simulation check

Simulate two regions with two workers each, capacity four, the observed arrival rate, and two routing modes: uniform versus hot-key locality. Add a persistent active import mode and record queue maxima and p95 waiting time across many seeds.

If the observed queue of 41 is common only under the hot-key burst model, H4 becomes plausible. If no reasonable parameter reproduces the tail without a code slowdown, investigate H1 or a hidden dependency state. The simulation is a sanity check on mechanism, not a replacement for the canary.

8. State the conclusion

A calibrated interim conclusion could be:

“The rollout has not demonstrated an overall latency improvement. Client median is 2 ms lower, but p95 and timeout rate are worse, with a 41-job West queue during a localized import. The server-span mean is invalid as a primary comparison because tracing overhead and missingness changed. The immediate safe action is to pause the import and keep the rollout at canary scale while comparing client timing, queue depth, and dependency wait. We need a matched high-load window before deciding on broad rollout.”

This conclusion supports an action without pretending that H1, H2, H3, or H4 has been fully isolated.

Failure Review

Check the dossier for these predictable failures:

• Formula without boundary: a mean or interval is reported without defining the event.
• Pooling across regimes: quiet and import periods are combined into one stationary baseline.
• Mean-only reasoning: a small average gain hides a tail or timeout regression.
• Independent-sample fiction: every request is counted as new evidence despite shared queue episodes and tenants.
• Instrument substitution: server spans are treated as user outcomes after their collection path changed.
• Simulation overreach: a model output is presented as proof that production behaves identically.
• Causal overclaim: a correlation or matched comparison is described as a confirmed mechanism.
• False certainty: uncertainty is removed from the conclusion instead of being assigned to the next measurement.

Trade-offs

There is no single perfect measurement plan. Client timing is closest to user experience but may hide internal causes. Server spans expose internal steps but can add overhead and missingness. Fine-grained queue counters reveal bursts but increase operational volume. A long canary improves evidence but delays a rollout decision.

The right trade-off is reversible action under explicit uncertainty. Pause the known import, preserve a small comparison group, and collect the signal that distinguishes the highest-cost hypotheses. Do not spend precision on a metric whose population is changing.

Evidence and Readiness

Use this rubric to judge the dossier:

A dossier can be incomplete and still be decision-ready if the action is reversible and the missing evidence is explicit. It is not ready when a polished number hides a broken boundary or missing population.

Before handing the dossier to another engineer, perform a red-team pass. Remove the title and conclusion, then ask whether the tables and traces still support the same hypotheses. Swap the order of the time windows and check whether the story changes. Recompute one summary after including timeouts and one after excluding the import period. If a tiny definition change reverses the recommendation, that sensitivity belongs in the conclusion. Finally, list the single observation that would most reduce uncertainty: an independent client timing, a queue trace, a matched canary, or a dependency counter. A model that identifies this observation is more useful than one that merely produces a narrower interval.

Final Challenge

Replace the indexing API with a scenario from your own technical work: a retry policy, a sensor alert, a batch pipeline, a recommendation metric, or a queue-backed service. Produce a one-page dossier containing:

1. the decision and operational boundary;
2. at least two competing hypotheses;
3. random variables and a small distribution or table;
4. a sampling and measurement audit;
5. a time, dependence, tail, or hidden-state concern;
6. one simulation or empirical sanity check;
7. a calibrated conclusion and reversible next action.

A strong dossier lets another engineer challenge one assumption without rebuilding the entire argument.

Resources

• [REFERENCE] NIST/SEMATECH e-Handbook: Exploratory Data Analysis - Focus: Use plots, robust summaries, and diagnostics to test the model before trusting a conclusion.
• [COURSE] MIT OpenCourseWare: Introduction to Probability - Focus: Revisit probability, sampling, conditional reasoning, and uncertainty as needed while building the dossier.

Key Takeaways

• Diagnose the decision, not just the metric: define the population, boundary, and failure cost first.
• Keep process variation, measurement error, missingness, sampling variation, time dependence, and hidden state distinct.
• Use comparisons, tails, queue traces, and small simulations to challenge a plausible story.
• Treat a conclusion as a calibrated action statement with explicit alternatives and limits.
• The best uncertainty model makes the next useful measurement obvious.