Device Driver Model, MMIO, DMA, and Interrupts

By the end of this lesson, you will be able to...

• Trace one block request from a kernel buffer to device completion.

• Explain the ownership transition that makes DMA safe.

• Diagnose an early buffer reuse, missing completion, or interrupt-context error.

Idea in one sentence: A driver turns a kernel request into device-visible descriptors and registers, then turns an asynchronous completion back into a safe kernel state transition.

Lesson 011 left a dirty cache buffer for device block 42. The buffer cache now asks the block driver to write it back. Calling the driver is not like calling memcpy: the device runs independently, reads descriptors from memory, transfers bytes with DMA, and signals completion later with an interrupt.

The driver must answer three questions precisely. Which registers tell the device where the request queue lives? Who owns the buffer while the device can read or write it? When an interrupt arrives, how does the kernel know which waiting request is complete without sleeping or touching a buffer that has already been reused?

Those are the driver's real interface. Register names and device manuals vary. The ownership and completion contract does not.

Core Insight

Use a small generic block-device model. The kernel has a fixed descriptor array in DMA-addressable memory. Each request contains a header, a pointer to the data buffer, and a one-byte status location. The driver also keeps a software table that maps a descriptor slot to the kernel buffer and any process waiting for it.

Plain meaning:

The driver hands the device a labeled package and keeps the receipt. While the device has the package, the CPU must not replace its contents or give the same package to somebody else. When the device rings the bell, the driver uses the receipt to return the package to the kernel.

In this scenario:

The dirty buffer for block 42 becomes the data portion of one descriptor chain. The driver records “slot 5 belongs to buffer 42” before it tells the device about slot 5. The buffer stays pinned until the completion path returns ownership.

Technical name:

Memory-mapped I/O (MMIO) exposes device control registers at addresses the CPU accesses with loads and stores. Direct memory access (DMA) lets the device read from or write to memory without the CPU copying each byte. A descriptor ring is a shared-memory queue whose entries describe DMA work. An interrupt transfers control to a short handler when the device has a completion or needs service.

The Naive Driver Corrupts Ownership

This unsafe sketch is tempting:

submit(device, buffer);
release_buffer(buffer);   // looks efficient, but the device may still use it

The cache may reuse buffer for block 99 before the device reads it. The device then writes bytes for block 99 into the storage location intended for block 42, or writes received data into memory now owned by another subsystem. The completion interrupt has no reliable object to wake or mark clean.

The opposite mistake is to make the interrupt handler wait for something else while holding the driver lock. Interrupt context is not a normal thread that can sleep. It must acknowledge the device, collect finished work, update short shared state, and wake or mark runnable any blocked requester. Heavier work can happen later in task context.

The driver therefore needs an ownership state machine:

CPU owns buffer
  -> CPU prepares descriptor and records request
  -> device owns/in-flight DMA buffer
  -> device marks completion and interrupts
  -> driver confirms completion
  -> CPU owns buffer again

At any point, exactly one owner is allowed to modify the buffer contents. “In flight” is not an empty status label: it is the reason cache eviction, reuse, and dirty-to-clean transitions must wait.

The Two Channels: Registers and Shared Memory

A driver usually communicates with a device through two different channels.

MMIO registers are not cached ordinary variables. A write can notify hardware or change a device state machine; reads can acknowledge status or observe a changing device value. The kernel maps and accesses them using the platform's rules. A device descriptor must be fully initialized before the driver updates the MMIO register or queue index that makes it visible to the device. On real architectures, the driver uses the required ordering primitives or DMA API so the CPU and device agree on that publication order.

DMA does not mean “any pointer is safe.” A user virtual address from a syscall is not automatically valid for a device. For this writeback path, the driver uses a kernel-owned cache buffer whose lifetime and DMA suitability are controlled. Production kernels may map DMA addresses, handle IOMMU translation, cache coherency, and device constraints; the teaching-kernel rule remains useful: never hand hardware a buffer whose ownership or lifetime is unclear.

Worked Trace: write back block 42

Assume the buffer cache holds dirty buffer B42, the driver has free descriptor slot 5, and a process may be waiting for a synchronous completion. The device uses a descriptor ring and an MMIO “notify” register.

Input: writeback(B42) for device block 42.

1. The cache gives the driver a stable request

The buffer cache locks or pins B42 so eviction cannot recycle it. It records that the buffer is in writeback rather than clean. The driver acquires its short queue lock and reserves slot 5.

State: B42 = dirty + pinned; descriptor slot 5 = reserved; device has not been notified.

2. The driver builds the shared descriptor chain

The driver fills entries in DMA-visible memory:

descriptor 5a: operation = WRITE, block = 42
descriptor 5b: address = DMA address of B42.data, length = 1024
descriptor 5c: address = request[5].status, length = 1
request[5]: buffer = B42, waiter = optional, in_flight = true

The header tells the device which block to write. The data descriptor identifies the buffer it may read. The status byte gives the device a place to record success or failure. The software request table is the driver's receipt: it links an eventual completion back to B42.

The driver performs the required memory-ordering publication step, then advances the ring's available index or writes the MMIO notify register. Only now may the device inspect slot 5.

State transition: B42: CPU-owned -> device-owned/in flight.

3. The driver releases the lock and the caller may block safely

The driver does not spin while the physical device works. It releases the queue lock. If the caller needs synchronous completion, it waits using the lock-and-wakeup discipline from lesson 009: publish the wait condition, release the appropriate lock, then sleep. The buffer remains pinned even if the requesting thread blocks.

The device independently reads the descriptors and DMA-reads B42.data. It may finish much later than the submitting CPU.

4. Device completion triggers an interrupt

After the device completes slot 5, it records a completion/status and raises an interrupt. Trap handling enters the driver's interrupt handler. The handler acknowledges the relevant device status, acquires the driver lock suitable for this context, and examines completed slots.

For slot 5, it finds request[5].buffer == B42 and reads the status. On success it clears in_flight, returns the descriptor chain to the free pool, and wakes the request's waiter or marks the buffer completion state visible. On failure it records an error; it must not lie by marking the buffer clean.

Output/decision: success lets the cache transition B42: writeback in flight -> clean and unpin it. Failure leaves the data needing retry, reporting, or recovery policy. In either case, CPU ownership returns only after the completion state is recorded.

5. The waiting path rechecks and continues

The awakened thread reacquires the relevant lock and rechecks request status; it does not assume that a wakeup itself proves success. It can now return an I/O result, retry, or let background writeback handle the next step.

Naive contrast: ringing the device before descriptor fields are visible lets hardware read partial or stale work. Releasing B42 before step 4 allows reuse during DMA. Marking it clean when merely submitted makes a power or device error look like durable storage.

So far, the kernel has transformed one cache-buffer request into a descriptor chain, hardware activity, an interrupt, and a completion state. The driver holds the same invariant throughout: slot 5, its DMA buffer, and its completion record describe one request until the device gives ownership back.

Trade-offs and Limits

DMA reduces CPU copying and lets the CPU schedule other work while a device transfers bytes. Interrupts avoid busy-waiting for infrequent completions. The trade-off is extra state: descriptor allocation, DMA-safe memory, publication ordering, locks shared with interrupt context, and explicit error recovery.

For a tiny or very fast operation, polling a status register can be simpler and avoid interrupt overhead. For many requests, rings and batched notifications improve throughput but make ownership and queue indices harder to inspect. A robust driver may switch between interrupt-driven and polling modes under load, but the basic ownership state machine remains the same.

This mechanism does not make hardware trustworthy or transfers infallible. A device can report errors, reset, or fail to complete. A driver needs timeouts, recovery, and resource limits before it can claim a production-quality interface. The signal to watch is a request that stays in_flight too long, a completion status error, or a growing count of pinned buffers: each means the ownership handoff did not finish normally.

Common Confusions

Confusion: “MMIO is just a function call through a pointer.”

Why it is tempting:

The source code often looks like a store to an address.

Better model:

The address represents hardware state. A read or write can communicate with a device and must follow the platform's ordering and access rules.

Confusion: “DMA lets the device use any buffer forever.”

Why it is tempting:

DMA bypasses per-byte CPU copying.

Better model:

The driver grants device access to a specific buffer for a defined interval. The buffer must remain valid and unmodified in incompatible ways until completion returns ownership.

Confusion: “An interrupt means the request succeeded.”

Why it is tempting:

The interrupt arrives after a device event.

Better model:

The handler must inspect completion status and map it to the matching request. An interrupt can indicate an error or several completions, not a universal success signal.

Confusion: “The interrupt handler can wait for a free descriptor.”

Why it is tempting:

The handler also needs shared queue state.

Better model:

Interrupt context should update short state and wake task-context code. Waiting while holding an interrupt-shared lock can deadlock progress or stall the CPU.

Check: The driver submits a DMA write for B42, then immediately unpins and recycles B42 for an incoming network packet. What is the specific failure risk?

Think first, then reveal.

Answer: The device can still DMA-read or DMA-write B42 while the network code changes or owns that memory. The transfer may write the wrong bytes, corrupt the new owner, and leave the completion mapping meaningless. The buffer stays pinned until completion returns ownership.

Check: An interrupt handler wakes a blocked writer, but the request status is still “in flight.” Should the writer report success?

Think first, then reveal.

Answer: No. The writer must recheck the status under the request's synchronization rule. A wakeup announces that relevant state may have changed; it does not substitute for a recorded successful completion.

Practice: review a DMA handoff

Review a proposed driver change: “To reduce latency, write the MMIO notify register first, then fill the descriptor fields and request table. Free the buffer as soon as notify returns.”

Write a short review that names:

1. two states that must be recorded before notification;
2. the ownership violation in freeing the buffer;
3. the correct point at which a dirty cache buffer may become clean; and
4. one action appropriate for the interrupt handler versus one action that belongs in task context.

Use this rubric:

Resources

• [BOOK] MIT 6.1810 xv6 book, interrupts and device drivers — Focus: trace a teaching kernel's interrupt path and driver ownership boundaries.
• [COURSE] MIT 6.1810 network-driver lab — Focus: inspect an E1000's DMA descriptor rings, MMIO control registers, and receive-buffer replacement.
• [DOC] Linux kernel Driver API — Focus: use the production-driver vocabulary for device discovery, DMA, interrupts, and lifecycle management.

Key Takeaways

1. A driver bridges synchronous kernel requests and asynchronous hardware using MMIO control plus DMA-visible shared memory.
2. A DMA buffer has an explicit ownership interval; it cannot be recycled until completion returns it to the CPU.
3. Descriptor, buffer, and software request-table state must be published before the device is notified.
4. Interrupt handlers record and signal completion; they do not assume success or block on long operations.