Performance Bottlenecks - Lock Contention & I/O Wait

By the end of this lesson, you will be able to...

• Distinguish CPU work from time waiting for a lock, pool, or external I/O resource.

• Trace how extra worker concurrency can turn one protected operation into a longer queue.

• Choose a bounded change that shortens a critical section or limits I/O pressure while preserving the protected invariant.

Idea in one sentence: When work waits for one scarce resource, more workers can create a longer line instead of more useful throughput.

Core Insight

For Atlas Shop, image-processing jobs begin to lag after a campaign adds new product images. The team increases the worker pool from 16 to 48. CPU stays around 37%, so the change appears safe. But completed jobs rise only slightly, queue lag grows, and job p99 climbs from 220 ms to 2.4 seconds.

The first explanation is “the workers need even more concurrency.” That works while each worker mostly owns its own CPU work. It fails when many workers must wait for the same mutex, connection pool, or remote dependency. CPU can be quiet because the missing time is spent waiting, not computing.

The stronger model is to name the queue in front of the scarce resource. Find what workers are waiting for, measure the wait, shorten or shard the protected work when safe, and cap I/O concurrency at a level the dependency can sustain. The aim is not to eliminate all waiting. It is to prevent unbounded waiting from hiding behind a low CPU number.

The Production Symptom: More Workers, Same Throughput

Atlas labels the job cohort before comparing worker counts:

job: product-image
input_size: medium
tenant: campaign
cache_state: miss
result: success

The values below are illustrative measurements from the same workload window:

The last number can exceed the ten-second wall interval because it is accumulated across waiting workers. Ten workers waiting for one second contribute roughly ten worker-seconds of wait. It is a pressure signal, not the duration of one job.

The CPU flame graph from the prior lesson is now unsurprising: it is not wide enough to explain the tail. The next evidence must concern waiting. In Go, a block profile records time blocked on synchronization primitives, and a mutex profile tracks contended mutexes; their stack traces identify the blocking or contention-related paths. Go runtime/pprof.

The Initial Model: Treat Moderate CPU as Free Capacity

Adding workers is a reasonable response when jobs are independent and the machine has idle CPU, memory, and downstream capacity. More ready work can then use processors that would otherwise be idle.

It becomes insufficient when every new worker reaches the same serialized gate. The system has more callers, but not more service capacity at that gate. They form a queue:

48 workers
  -> one shared index mutex
  -> one metadata connection pool
  -> one remote metadata service

The first queue can make the second harder to see. A worker stalled on a mutex has not reached the metadata pool yet. After the mutex is improved, the I/O pool may become the next visible boundary. That does not mean the first fix was wrong. It means the bottleneck moved, so the resource question must move too.

Low CPU is therefore a condition to investigate, not a capacity verdict. Lock wait, pool wait, connection saturation, scheduler delay, and external response time can all dominate a job's wall-clock duration while the processor has spare cycles.

The Failure Mechanism: I/O Hidden Inside a Critical Section

Each image job has a small CPU resize step. Then it updates a shared in-process index that says whether a product image is ready. The original path is simplified here:

lock indexMu
  read current image state
  request metadata from remote store
  write final state into shared index
unlock indexMu

The lock protects an invariant: a job must not publish a ready image with stale metadata. Keeping that invariant is correct. The failure is holding the one lock while making a remote request that may take tens of milliseconds.

At the runtime level, a blocked lock path may eventually wait using a kernel synchronization mechanism. Linux futexes provide a way to wait for a condition in shared-memory synchronization and wake waiters later. That fact helps explain why the process is not using CPU while blocked; it does not explain which application invariant justified the lock. Linux futex(2).

The evidence earns a narrow conclusion: indexMu is an avoidable serialization point because slow I/O occurs while it is held. It does not prove that all locks are bad, or that the remote store is healthy.

Mitigation: Preserve the Invariant, Shorten the Gate

Atlas changes the state transition in small steps:

lock key-specific state
  read current version
  mark this version as metadata-pending
unlock

request metadata with a bounded I/O permit

lock key-specific state
  commit only if the version is still current
unlock

The first lock is short. The metadata request happens outside it. The final version check prevents an older job from overwriting a newer product-image state. A per-key in-flight marker or coalescing rule is also needed when two jobs for the same image arrive together; otherwise moving the request outside the lock may create duplicate remote calls.

The second control is a bounded metadata pool of eight permits. It does not make the remote service faster. It makes the admission boundary explicit: at most eight metadata requests are in flight, while other jobs remain visible in a queue with a timeout and backpressure policy. A readiness-based I/O design can wait for multiple file objects to become ready without dedicating one blocking thread to each one, but it still cannot remove dependency latency or its capacity limit. Python selectors.

This is a situated preference, not a universal recipe. Key-specific state and a bounded pool fit when the invariant can be expressed per image and the remote service has a known safe concurrency range. A global order, an atomic transaction, or a different ownership boundary may be required for another invariant.

Signals to Watch After the Change

Atlas repeats the same labelled workload and changes one boundary at a time:

These illustrative results support, but do not prove forever, that the new boundary is better. The pool wait is a signal to watch: if it rises while remote latency rises, adding permits may only overload the dependency. If it stays low but queue lag grows, another resource or admission limit needs investigation.

So far, the team has turned a vague “low CPU but slow jobs” incident into two named queues: a mutex queue that was made short, and an I/O queue that is deliberately bounded. It has not made the remote store reliable, eliminated the need for retries, or shown that a larger worker fleet is always safe.

Trade-offs and Limits

Shorter critical sections improve concurrency, but they cost a more explicit state machine, version checks, and duplicate-work control. A bounded I/O pool protects a dependency, but it introduces its own queue and may increase individual wait during a burst. The trade-off is between uncontrolled contention and visible, policy-governed admission.

Avoid these false fixes:

• Removing a lock without replacing the invariant can create lost updates or publish stale results.
• Raising the worker count without changing the constrained resource can amplify mutex or pool wait.
• Switching to asynchronous I/O can reduce blocked-thread use, but it does not make a slow dependency respond faster.
• Sharding state can reduce contention, but poor key distribution can leave one shard hot.

The boundary signal is a mismatch between the intervention and the next measurement: low mutex wait with high I/O wait, stable I/O wait with rising queue lag, or lower latency with state-correctness failures. Each calls for a different response; none is evidence to tune arbitrary CPU code.

Check: After shortening a critical section, CPU is still 35%, mutex wait is low, and metadata-pool wait dominates job p99. Should the team add 40 more workers immediately?

Think first, then reveal.

Answer: No. The queue has moved to the metadata dependency. First inspect its latency, permit limit, timeout behavior, and safe concurrency. More workers may only make the I/O queue longer or overload the dependency.

Practice: Choose the Next Queue to Investigate

A report-generation service has 24 workers. CPU is 28%, p99 is 3 seconds, and a mutex profile is quiet. The database connection pool has eight slots; its wait p95 is 1.1 seconds, while database execution p95 is 70 ms. Each report makes four serial metadata queries.

Propose the smallest next investigation and one bounded mitigation. Name the resource evidence, the likely queue, the correctness concern, and the result that would make you revise the plan.

A good answer should mention:

• pool-wait and dependency evidence rather than a CPU optimization, because execution is short while admission wait is high;
• the eight-slot database pool or serial round trips as the likely queueing boundary;
• a bounded batching, query-shaping, or concurrency change that preserves report completeness and isolation requirements;
• before-and-after pool wait, query count, p95/p99, errors, and report-correctness checks; and
• a revision signal such as a slower query plan, a saturated database, or a new mutex queue after the change.

Connections

The profile lesson taught that high latency can be waiting rather than CPU. The flame-graph lesson taught how to read the selected profile without confusing it with time order. This lesson makes that waiting actionable: locate the queue, preserve the invariant, and make admission visible. The next review lesson combines cache reuse, profiling, stack evidence, and waiting boundaries into complete optimization decisions.

Resources

• [DOCS] Go runtime/pprof — Focus: Compare CPU, block, mutex, heap, and allocation profiles; inspect what block and mutex samples represent.
• [DOCS] Linux futex(2) — Focus: Connect blocked shared-memory synchronization to a lower-level waiting primitive without confusing it with the application invariant.
• [DOCS] Python selectors — Focus: Examine readiness-based I/O multiplexing and the distinction between blocked threads and dependency latency.
• [ARTICLE] Linux perf Examples — Focus: Explore event-oriented evidence for CPU, I/O, scheduler, and off-CPU questions.

Key Takeaways

• Low CPU does not imply spare useful capacity: workers may be queued behind a lock, pool, or external dependency.
• More concurrency helps only when the constrained resource can serve the extra work; otherwise it lengthens the waiting line.
• A lock protects an invariant. Improve its critical section without discarding the state rule it enforces.
• Bounded I/O concurrency turns hidden overload into an observable admission policy, but it does not remove dependency latency.
• Measure mutex wait, pool wait, throughput, tail latency, errors, and state correctness together after each bounded change.