The Storage I/O Path from Syscall to Device or Service

By the end of this lesson, you will be able to...

• Trace a storage read or write across application, cache, kernel or client, queue, and durable-service boundaries.

• Identify where an operation may return early, wait, retry, or become durable.

• Select evidence that distinguishes a cache, metadata, queue, device, network, or remote-service delay.

Idea in one sentence: Storage latency belongs to a path of handoffs and queues, so useful diagnosis asks where a request is waiting and what each layer has promised—not which single box is “slow.”

The upload that says “complete” while work continues

An editor uploads a new course package. The application writes the package to a local staging file, verifies it, and sends it to object storage. The user sees Upload complete, but background work still hashes data, drains dirty pages, sends network requests, updates object metadata, and invalidates a cache.

Later, a learner opens the course. The page is slow. Someone says, “The disk is slow.” Another person says, “The object store is slow.” Both may be right, but neither statement identifies the request that the learner is waiting for.

The visible operation has crossed several boundaries. Application code may wait on a runtime pool. A syscall may find bytes in the page cache or trigger a deeper filesystem read. A file client may resolve metadata before it requests data. A network client may queue a remote upload. A device or remote service may acknowledge one stage while later work continues elsewhere.

The important shift is from storage as a box to I/O as a path. A path has owners, copies, queues, and promises. Once that path is visible, “slow storage” becomes a set of testable questions.

Core Insight

Plain meaning:

A request moves through several people or components. At each handoff, it can finish, wait, be copied, be queued, or be sent deeper.

In this scenario:

Reading the course package can finish from an application cache, the operating system’s page cache, a local device, or a remote object service. Writing it can return after buffering, after a durable local journal decision, or after a remote service’s acknowledgement, depending on the contract.

Technical name:

The ordered handoffs are the storage I/O path. Each handoff is a boundary where ownership, visibility, queueing, and durability may change.

Do not treat the diagram below as one literal implementation. A local file read and a remote object read take different branches. The model is useful because it names the questions that survive those differences.

application
  → runtime / client buffer
  → syscall or storage API
  → local cache or client cache
  → namespace and metadata lookup
  → filesystem / object client
  → queue and transport
  → local device OR network + remote service
  → bytes or acknowledgement return upward

At every arrow, ask four things:

1. Who owns the next state?
2. Can this layer return a cached answer or acknowledge early?
3. Can requests queue, retry, or be throttled here?
4. What evidence would show that this boundary is the wait?

A read trace: from file descriptor to bytes

Suppose the package verifier reads a local staging file. The input is a normal file read:

read(fd, buffer, 128 KiB)

The application expects bytes. It does not need to know whether the kernel already has them in memory or must fetch them from a device. That distinction changes the timing, though.

This trace has two legitimate fast paths. On a page-cache hit, steps 4–6 may not be on the critical path at all. On a cache miss, they are. A fast read from memory is good news for the caller, but it is not proof that the device or remote service is healthy. It only proves that this read avoided them.

The naive failure is to measure only average read() duration and call the device healthy. If a cache hides 99 fast reads while the hundredth miss waits ten seconds behind a queue, users who take that miss experience a very different system. The path, not the average label, explains the symptom.

Check: A file read is slow only after the host restarts; repeated reads of the same file become fast. Which boundary is the first strong suspect?

Think first, then reveal.

Answer: The page-cache boundary. Restart removes cached pages, so the first read likely takes the deeper metadata and device or service path while later reads hit memory. Confirm with cache-miss, device, and metadata timing rather than assuming the application changed.

A write trace has two clocks

Writes are harder because there is often a clock for when the caller may continue and another for when the final bytes are materialized in stable storage. The two clocks may be close together, or they may be deliberately separated by buffering and writeback.

Consider the staging-file write before the upload starts:

write(fd, package_chunk, 1 MiB)

application buffer
   → syscall
   → page cache marks pages dirty
   → write may return to application
   → later writeback / journal ordering
   → lower queue
   → device acknowledgement

In a common buffered path, write() may return after the kernel accepts bytes into dirty cached pages. That can improve throughput by batching and reordering work. It does not by itself make a power-loss promise. A stronger operation—often an explicit synchronization step, or a higher-level transaction and journal contract—asks the system to cross a defined durability boundary before returning success.

The previous lesson explained the recovery side: a durable journal decision can support correct restart even while ordinary data pages are still being written back. The I/O-path view adds a practical question: where is the operation now, and which later queue must drain before the relevant durability condition is met?

The exact rows vary by API. That is the point: do not use a return value without naming the contract behind it.

One remote branch: object upload

The course package is eventually uploaded to object storage. This path does not necessarily use the host filesystem after the upload client reads the staging file. It has a different set of boundaries:

upload worker
  → client-side multipart buffer
  → DNS / connection pool / TLS session
  → network socket send queue
  → network path and retries
  → object-service request handling
  → object metadata and durable data path
  → service acknowledgement

The durable authority is now remote. A client-side “chunk accepted” event is not automatically the same as “the complete object is published under its final key.” Many systems distinguish part uploads, a final completion request, metadata visibility, and later cache invalidation. The team must trace the API’s own acknowledgement points just as carefully as it traces a local fsync boundary.

This branch also explains why network metrics alone are insufficient. A low RTT can coexist with long remote-service queueing. A fast service response can coexist with a client connection pool that has no available slot. An upload can be quick while a later read is slow because metadata lookup or cache expiry has changed. I/O paths are compositions, not a contest between “network” and “storage.”

Check: An object upload’s network RTT is normal, but end-to-end upload time grows as workers increase. Which two boundaries deserve immediate inspection?

Think first, then reveal.

Answer: Inspect the client-side worker/connection-pool and socket-send queues, then the remote service’s request throttling or queueing signals. Normal RTT measures a network trip, not time waiting before a request is sent or after the service receives it.

Queues turn small waits into tail latency

Each layer may be quick when idle. Queues make the combined path slow under pressure. A runtime can queue requests for a worker. A filesystem can wait on metadata. Dirty-page limits can throttle a writer. A device or remote service can queue I/O. A retry can place a previously slow request back into the path and add more work.

arrival rate > service rate for a boundary
        → queue grows
        → waiting time grows
        → timeouts and retries may add more arrivals

The first remedy is not automatically “increase concurrency.” More concurrency can increase throughput when a path is underused, but it can also deepen queues and amplify contention when a lower boundary is saturated. This is the central trade-off: buffering and concurrency hide short waits and improve utilization; they also make ownership and tail latency harder to see when work accumulates.

For a useful incident trace, record one request across boundaries:

09:00:00.000  worker acquired
09:00:00.002  syscall entered
09:00:00.003  page-cache miss
09:00:00.004  metadata lookup complete
09:00:00.006  lower I/O submitted
09:00:00.406  lower I/O completed
09:00:00.407  bytes returned

This trace does not require a device-driver implementation. It tells you that 400 ms is below the metadata boundary, after submission, before completion. The next diagnostic question can be precise: lower queue depth, device latency, remote-service request time, or retry count—not “why is storage slow?”

Trade-offs and limits

Layering improves reuse and separation of concerns. Caches reduce deep reads, buffering batches writes, queues absorb bursts, and remote services can provide a different durability contract. The cost is that a simple user operation may have several partial completion points.

• A cache hit improves latency but hides the health of the deep path.
• Buffered writes improve throughput but separate write() return from durable completion.
• Queues smooth short bursts but create tail latency and can trigger retries under sustained overload.
• A remote service centralizes durability but adds client, network, and service boundaries to every operation.

The I/O-path model does not identify a root cause by itself. It locates the waiting boundary and suggests evidence. Device scheduling, DMA, interrupts, driver behavior, and detailed block-layer algorithms are intentionally outside this track; they are valid next steps only after the path tells you they matter.

Common confusions

Confusion: Every slow read is a disk problem

Why it is tempting:

The user asked for stored bytes, so the physical device feels like the obvious owner.

Better model:

The request may be waiting for a runtime worker, a cache miss, metadata lookup, a lower queue, network retry, or remote-service throttle. Trace the request before naming the layer.

Confusion: A successful write() means durable storage finished

Why it is tempting:

The API reports success with a single return value.

Better model:

Success means the contract of that call was met. Buffered acceptance, a journal decision, lower-page writeback, and remote-service durability are different boundaries that must be named separately.

Confusion: More concurrency always makes I/O faster

Why it is tempting:

Parallel requests can hide latency when capacity is unused.

Better model:

Once a lower layer is saturated, additional concurrent work grows queues and tail latency. Change concurrency while watching queue age, completion rate, retries, and errors.

Practice: annotate the missing boundary

A course player sometimes takes eight seconds to start a newly uploaded video. Existing popular videos start in under 100 ms. The player trace shows that it obtains the video key immediately, then waits 7.7 seconds after sending the first object request. CDN hits are near zero for the new key; network RTT is 20 ms; remote-service request latency is high; older video keys are normal.

Write the next diagnostic plan. Name the likely path segment, two competing explanations, one safe mitigation, and the evidence that would distinguish them.

Model answer: The wait is below the player’s key lookup and after the outgoing object request, so investigate the object-service request and origin path rather than the page cache or local metadata path. Competing explanations include cold-origin retrieval or service-side queueing/throttling for newly uploaded objects. A safe mitigation is to warm or prefetch the newly published object before exposing it broadly, while keeping a fallback and avoiding a large retry burst. Compare service request duration, response status or throttling signals, origin retrieval metrics, object availability state, and request concurrency. The low CDN hit rate explains why the deep path is exposed; it does not by itself identify which deep boundary is slow.

Resources

• [BOOK] Operating Systems: Three Easy Pieces — Focus: Connect syscalls, caching, filesystems, and I/O completion without treating them as one box.
• [DOC] Linux block layer documentation — Focus: Use this only when a trace has already localized the wait below the filesystem layer.
• [DOC] AWS S3 multipart upload overview — Focus: Compare client part acceptance, completion, and object publication boundaries in a remote object path.

Key Takeaways

1. A storage operation crosses owners, caches, queues, and durability boundaries; an end-to-end trace makes the relevant one visible.
2. Reads may finish at a cache or wait for metadata and lower I/O; writes often have different acceptance and durable-completion clocks.
3. Diagnose the boundary where work waits or retries, then use its specific signals instead of attributing every delay to “the disk.”