Deferred Disk I/O: Serving 40GB Through a Virtual NVMe

I needed to serve a full 40GB Windows installation to a guest VM that only had 4GB of RAM mapped. The solution was a deferred I/O worker thread -- the hot regions live in RAM, everything else gets read on demand from the real disk via ZwReadFile. Getting this right was the difference between a guest that boots and a guest that corrupts its own memory.

The problem is straightforward. My guest VM runs a full Windows 10 installation on a 40GB NVMe disk image. I can't map 40GB into the guest's physical address space -- the host machine only has 32GB of total RAM and Windows is already using most of it. I had to be selective about what lives in memory and what gets fetched on demand.

The 4GB memory map

I analyzed Windows boot I/O patterns to figure out which disk regions get accessed during boot. The answer: mostly the first 2GB (boot files, system32, registry hives) and a chunk in the 2-4GB range (paging files, additional drivers). I pre-load these two regions into guest RAM. Everything else -- user profiles, program files, the rest of the 40GB -- is served on demand.

// pre-loaded guest RAM regions
const REGION_HEAD: (u64, u64) = (0x0000_0000, 0x8000_0000);  // 0-2GB
const REGION_MID:  (u64, u64) = (0x8000_0000, 0x1_0000_0000); // 2-4GB

// everything else: deferred to worker thread
// 4GB-40GB served on demand via ZwReadFile

fn init_guest_disk(image_path: &str) -> DiskState {
    let file = open_disk_image(image_path);
    let head = allocate_contiguous(0x8000_0000);
    let mid  = allocate_contiguous(0x8000_0000);

    // read the first 4GB from disk into RAM
    read_file_into(file, 0, head, 0x8000_0000);
    read_file_into(file, 0x8000_0000, mid, 0x8000_0000);

    DiskState { file, head, mid, total_size: 40 * 1024 * 1024 * 1024 }
}

The 4GB pre-load takes about 8 seconds at startup. That's acceptable because it only happens once. After that, reads from the first 4GB are instant memory copies -- no disk latency at all.

The worker thread

For reads beyond 4GB, I can't service them synchronously in the #VMEXIT handler. ZwReadFile can block, and I'm running at DISPATCH_LEVEL inside the exit handler on the guest's core. Calling a blocking function there would deadlock the system.

So I built a deferred I/O system. The exit handler queues the read request and pauses the guest vCPU. A dedicated worker thread running on a host core picks up the request, calls ZwReadFile at PASSIVE_LEVEL, copies the data into the guest's buffer, and signals the vCPU to resume.

struct IoRequest {
    disk_offset: u64,
    length: u32,
    dest_gpa: PhysAddr,
    completion: KEVENT,
}

fn io_worker_thread(state: &DiskState) {
    loop {
        // wait for a request
        let req = dequeue_request();

        // read from the real disk (PASSIVE_LEVEL, can block)
        let buf = allocate_temp_buffer(req.length);
        ZwReadFile(
            state.file,
            NULL,          // no async event
            NULL,          // no APC
            &io_status,
            buf,
            req.length,
            &req.disk_offset,
            NULL,
        );

        // copy data to guest physical address
        copy_to_guest_pa(req.dest_gpa, buf, req.length);

        // signal the vCPU to resume
        KeSetEvent(&req.completion, 0, FALSE);

        free_temp_buffer(buf);
    }
}

The NVMe command handler

The guest's NVMe driver sends read commands through MMIO to the virtual NVMe controller I implemented. When a read command arrives, I extract the LBA (Logical Block Address) and sector count, convert to byte offsets, and decide whether it's a fast-path (in RAM) or slow-path (deferred) operation.

fn handle_nvme_read_cmd(
    sqe: &NvmeSubmissionEntry,
    state: &DiskState,
) -> NvmeCompletion {
    let lba = sqe.cdw10 as u64 | ((sqe.cdw11 as u64) << 32);
    let count = (sqe.cdw12 & 0xFFFF) + 1;
    let byte_off = lba * 512;
    let byte_len = count as u64 * 512;
    let dest_gpa = sqe.prp1;

    if byte_off + byte_len <= REGION_MID.1 {
        // fast path: copy from pre-loaded RAM
        let src = if byte_off < REGION_HEAD.1 {
            state.head.add(byte_off as usize)
        } else {
            state.mid.add((byte_off - REGION_MID.0) as usize)
        };
        memcpy_to_guest(dest_gpa, src, byte_len as usize);
        NvmeCompletion::success()
    } else {
        // slow path: deferred I/O
        queue_deferred_and_wait(byte_off, byte_len, dest_gpa);
        NvmeCompletion::success()
    }
}

The fast path is a straight memcpy -- sub-microsecond for typical 4KB reads. The slow path adds disk latency (typically 50-200 microseconds for my NVMe SSD) plus the thread scheduling overhead. During Windows boot, about 95% of reads hit the fast path. The remaining 5% are slow-path reads for files that happen to live beyond the 4GB boundary.

The write bounds bug

This one was subtle and it cost me a full day. The guest writes data back to disk during boot -- registry transaction logs, pagefile initialization, and so on. My write handler had a bug in the address calculation.

// THE BUG:
fn handle_nvme_write_cmd(sqe: &NvmeSubmissionEntry, state: &DiskState) {
    let lba = sqe.cdw10 as u64;
    let byte_off = lba * 512;
    let src_gpa = sqe.prp1;

    // BUG: I was using byte_off as the destination pointer offset
    // directly into the mapped region WITHOUT subtracting the
    // region base address
    let dest = state.head.add(byte_off as usize);
    memcpy_from_guest(src_gpa, dest, byte_len);
    // writes to LBA in the mid region (2-4GB) went to
    // head_ptr + 2GB+ = WAY past the allocated buffer
}

// THE FIX:
fn handle_nvme_write_cmd(sqe: &NvmeSubmissionEntry, state: &DiskState) {
    let byte_off = lba * 512;
    if byte_off < REGION_HEAD.1 {
        let dest = state.head.add(byte_off as usize);
        memcpy_from_guest(src_gpa, dest, byte_len);
    } else if byte_off < REGION_MID.1 {
        // subtract mid region base to get offset within mid buffer
        let dest = state.mid.add((byte_off - REGION_MID.0) as usize);
        memcpy_from_guest(src_gpa, dest, byte_len);
    } else {
        // beyond 4GB: queue to worker for disk write
        queue_deferred_write(byte_off, byte_len, src_gpa);
    }
}

The bug: when the guest wrote to an LBA in the mid region (2-4GB), I added the full byte offset to the head pointer. The head buffer is only 2GB. So a write to byte offset 2.5GB would go to head_ptr + 2.5GB, which is 500MB past the end of the head buffer. This overwrote whatever kernel memory was allocated after it -- pool headers, other driver allocations, random kernel data structures.

The corruption was silent. No immediate crash. The writes landed in kernel pool memory that wasn't being actively used at that moment. It manifested minutes later during boot when the kernel tried to read a corrupted pool allocation and got garbage. The BSOD was BAD_POOL_HEADER -- a classic sign of out-of-bounds writes.

Performance characteristics

With the deferred I/O system working correctly, the Windows boot takes about 45 seconds. The breakdown: roughly 12 seconds for pre-loading the 4GB at startup, 8 seconds for the actual boot sequence (UEFI to login screen), and 25 seconds of deferred I/O stalls where the guest is waiting for disk reads. Most of those stalls are loading drivers and services that live outside the pre-loaded region.

If I pre-loaded more RAM -- say 8GB or 16GB -- the boot would be significantly faster. But even at 4GB, it works. The guest boots to a login screen, the desktop loads, applications run. The deferred I/O system keeps serving reads for as long as the guest is alive. It's not fast, but it's functional, and it proved that the whole architecture -- blue-pill hypervisor, virtual NVMe, deferred I/O worker -- can actually boot a real operating system.