Optimization Case Studies - Real Production Systems

By the end of this lesson, you will be able to...

• Reconstruct an optimization decision from a symptom, a resource question, and comparable evidence.

• Link cache reuse, profiling, flame graphs, and waiting queues without treating any one tool as a universal fix.

• Defend the next bounded change in a request or worker incident, including its correctness and failure signals.

Idea in one sentence: Effective optimization changes the largest justified source of work first, then measures what becomes the next limit.

Core Insight

For Atlas Shop, an expensive request and a slow worker queue are one kind of problem: identify the affected work, bound avoidable work, and follow the evidence to the next scarce resource.

What You Can Now See

Atlas Shop has two campaign incidents: a slow homepage cache miss and image-worker queue lag. Familiar fixes are TTL, database tuning, serializer replacement, or more workers.

Those fixes work only when they match the dominant cost. A high aggregate cache hit ratio can hide an expensive route; a quiet CPU profile can hide mutex waiting; a narrower CPU branch can coexist with unchanged p99 because I/O is now the boundary.

The review model is a loop:

name the affected work class
-> remove work that should not happen
-> identify the resource used by work that remains
-> change one bounded mechanism
-> compare user impact, resource evidence, and correctness
-> name the next boundary

This is not a fixed order of tools. It is a way to keep a cache change, a profile, and a concurrency limit connected to the evidence that justifies each one.

The Concepts Together

Each row corrects a different shortcut. Caching does not prove the origin path is fast. A profile does not decide a product contract. A flame graph is not a timeline. More workers do not create more capacity at a one-at-a-time resource.

Common Confusions

Confusion: “The cache hit ratio is high, so the request path is healthy.”

Why it is tempting: One percentage is easy to read and small static assets may dominate it.

Better model: Break cache evidence down by route or object class, then pair it with origin work and user tail latency. A safe key can reduce repeated work; an unsafe key can serve the wrong response.

Confusion: “The widest frame is the fix.”

Why it is tempting: The frame is visibly large and has a recognisable function name.

Better model: In a standard flame graph, width is aggregate stack population. Follow callers and children to find the decision that creates the work. The leaf may be costly, but it may also be the endpoint of an oversized input.

Confusion: “CPU below 100% means we can add workers.”

Why it is tempting: CPU looks like a universal capacity meter.

Better model: Name the queue. Workers can wait at a mutex, connection pool, scheduler, or remote service. More concurrency helps only if the constrained resource can serve the additional work.

Synthesis Example: Two Incidents, One Evidence Loop

Case A: The route that should not reach origin so often

Atlas sees a 94% aggregate CDN hit ratio. Yet /home cache misses have a 600 ms p99 and origin CPU remains high. The first wrong move would be to tune JSON encoding because it appears in a source search.

The team first separates the route from the aggregate. The homepage has only 45% edge hits because tracking parameters and an irrelevant device header fragment the key. Spanish and English output genuinely differ; the tracking parameters do not change the public HTML.

The bounded change is home|language, after output tests verify the sharing boundary. A shield or request coalescing layer can then consolidate equivalent refill work. This improves reuse, but it costs an explicit cache contract and does not make private or personalised responses shareable. The relevant correctness signal is wrong-language or stale-content reports, not just a larger hit ratio. HTTP caching guidance explains why variation and freshness rules matter before merging request classes. MDN HTTP caching.

After reuse improves, the remaining cache misses matter more. The team captures a CPU profile for the same route, cache state, and campaign cohort. Response construction and encode_json are wide. The flame graph shows that most encoder samples arrive through build_catalog_payload; it does not show that the encoder happened last. The team reduces unused variant rows, then compares CPU-path width, cache-miss p99, and response fields.

The result is a better origin path, not a declaration of victory. If p99 remains high after the CPU branch shrinks, the next question is waiting or dependency cost, not another arbitrary serializer change.

Case B: The workers that create a longer line

Image workers have a p99 of 2.4 seconds. Increasing the pool from 16 to 48 leaves CPU near 37%, barely changes throughput, and raises queue lag. A mutex profile shows substantial wait around record_result.

The first wrong move would be to add still more workers. The workers are not starved of CPU; they are queued behind a state transition that holds a shared lock while requesting remote metadata.

The bounded change makes the invariant visible: mark a version as pending under a short key-specific lock, make metadata I/O outside the lock under a bounded permit pool, then commit only if the version is still current. The important trade-off is extra state-machine and duplicate-work control in exchange for less uncontrolled contention. A mutex profile can report cumulative wait across several waiters, which is why the metric must be read alongside job p99, throughput, and correctness. Go runtime/pprof.

After mutex wait falls, metadata-pool wait may become visible. That is the expected next boundary. The correct response is to measure dependency latency and safe in-flight capacity, not to assume that more workers will erase I/O waiting.

So far, both cases have followed the same loop. One removes unnecessary executions before inspecting a remaining miss. The other exposes a queue before changing concurrency. Neither optimizes a component because it is famous or easy to edit.

Retrieval Check

1. Why can an aggregate cache hit ratio be high while origin p99 is still poor?
2. What does a standard flame graph's width represent, and what does its x-position not represent?
3. Name two signals that distinguish CPU work from waiting work.
4. Why can a bounded I/O pool be an improvement even though it introduces a queue?

Answers: A high ratio can be dominated by cheap objects while an expensive route misses. Width represents aggregate stack population for the captured profile; x-position is not time order. CPU samples plus mutex/block/pool-wait evidence distinguish resource hypotheses. A bounded pool makes admission and overload visible, protects a dependency's safe concurrency range, and creates a queue that can be measured and controlled.

Transfer Challenge

An order-confirmation API has a 98% cache hit ratio overall. The POST /confirm path is not cacheable, has a 1.8 second p99, CPU is 25%, and database execution is 90 ms. A block profile shows long waits obtaining a connection from a 12-slot pool; the service has 80 request handlers. The team proposes adding 80 more handlers.

Review this proposal. State the next resource question, one bounded mitigation to test, the correctness constraint, and the before-and-after evidence required for approval.

A good answer should mention:

• the affected non-cacheable POST /confirm cohort rather than the aggregate cache ratio;
• connection-pool admission and dependency capacity as the next resource question, not a cache or CPU fix;
• a bounded change such as reducing unnecessary serial queries, choosing a safe pool/admission limit, or applying backpressure rather than doubling handlers;
• order confirmation's exactly-once or idempotency contract, timeout behaviour, and safe retry semantics; and
• comparable p95/p99, pool wait, database execution, queue depth, errors, duplicate confirmations, and successful-confirmation evidence.

What Comes Next

This review closes the performance-evidence cluster: classify the work, choose the evidence, change one boundary, and verify the new steady state. The next lessons harden the full path against bursts: request coalescing prevents a shared cache miss from multiplying origin work, then rate limits and worker recovery make admission and failure policy explicit.

Resources

• [ARTICLE] Flame Graphs — Focus: Recheck what width, stack depth, color, and x-position mean before making a graph-driven change.
• [DOCS] Grafana Pyroscope — Focus: Compare profiles with metrics, logs, traces, and incident windows instead of treating a screenshot as proof.
• [ARTICLE] HTTP caching — Focus: Revisit freshness and variation before merging equivalent public requests.
• [DOCS] Go runtime/pprof — Focus: Contrast CPU, block, and mutex evidence when an incident may be computation or waiting.

Key Takeaways

• Optimize a labelled work class, not an attractive global metric or familiar component.
• Removing repeated work can reveal the real cost of the work that remains; that is progress, not evidence that the first change failed.
• Use profiles and flame graphs to form a resource hypothesis, then test one bounded structural change.
• Waiting queues require an ownership or admission decision; low CPU alone does not establish spare useful capacity.
• Approve an optimization only when user impact, resource evidence, and correctness all improve or remain within named limits.