Booting Windows Under a Custom Hypervisor

I tried to boot Windows 10 as a guest VM under my hypervisor. The boot sequence got all the way to "Loading files..." with the progress bar at 100%, then crashed with 0xC0000225. Getting from that crash to a working boot took six separate fixes, each one found through hours of WinDbg + serial debugging. This is the story of all six.

The setup: QEMU with WHPX acceleration, running my hypervisor as L0, Windows 10 22H2 as the guest. A 40GB virtual NVMe disk backed by a raw image file. OVMF firmware for UEFI boot. The guest has 8GB of RAM, 4 vCPUs, and no idea it's virtualized.

Fix 1: GDT patching (#GP elimination)

The first crash was immediate -- a #GP (General Protection Fault) during the transition from bootmgfw to winload. The Windows boot loader switches to its own GDT, and my hypervisor was still using the OVMF GDT. When the guest loaded a segment selector that didn't exist in the old GDT, the CPU faulted.

The fix: intercept GDT switches. When the guest writes to GDTR (via LGDT), I update my shadow GDT to match. I don't intercept LGDT directly -- that would be expensive. Instead, I handle it lazily: when a #GP fires on a segment load, I check if the guest's GDTR has changed, resync, and retry the instruction.

fn handle_gp(vmcb: &mut VMCB, regs: &mut GuestRegs) {
    let guest_gdtr = vmcb.save.gdtr;
    let shadow_gdtr = get_shadow_gdtr();

    if guest_gdtr.base != shadow_gdtr.base
    || guest_gdtr.limit != shadow_gdtr.limit {
        // guest loaded a new GDT -- resync
        resync_gdt(vmcb, guest_gdtr);
        // don't advance RIP -- retry the instruction
        return;
    }
    // real #GP -- inject back to guest
    inject_exception(vmcb, GP_VECTOR, vmcb.control.exit_info_1);
}

Fix 2: MTRR shadow

Memory Type Range Registers define caching behavior for physical memory regions. The guest reads MTRRs to decide whether memory is write-back, write-through, uncacheable, etc. My hypervisor wasn't emulating MTRRs at all -- the guest was reading whatever the host's MTRRs were, which didn't match the guest's physical memory layout.

Winload uses MTRR state to configure page table caching attributes. With wrong MTRRs, it was setting up pages as uncacheable that should have been write-back. Performance was abysmal and certain memory-mapped regions broke entirely.

The fix: shadow all MTRR MSRs. I maintain a set of virtual MTRR values that present a sane memory map to the guest -- all RAM as WB, MMIO regions as UC.

Fix 3: Deferred disk I/O

The guest has a 40GB NVMe disk. I can't load 40GB into RAM. The original approach loaded 4GB -- the first 2GB and a middle 2GB chunk -- and hoped the boot process wouldn't need anything else. It did.

// deferred I/O: worker thread serves disk reads via ZwReadFile
fn handle_nvme_read(lba: u64, count: u32, buf: *mut u8) {
    let offset = lba * 512;
    let size = count as u64 * 512;

    if is_cached(offset, size) {
        memcpy_from_cache(buf, offset, size);
    } else {
        // queue to worker thread -- reads from the raw image file
        queue_deferred_read(DiskIoRequest {
            offset,
            size,
            buffer: buf,
            event: create_event(),
        });
        // guest vCPU halts until the read completes
        halt_vcpu_until(request.event);
    }
}

The worker thread runs on a host core, calling ZwReadFile against the raw disk image. When the guest vCPU issues an NVMe read for an uncached sector, I pause the vCPU, queue the I/O, and resume it when the data arrives. This way the full 40GB disk is accessible, but only 4GB sits in RAM at any time.

Fix 4: Write bounds checking

The guest was writing beyond the allocated NVMe buffer. I had bounds checking for reads but not writes. Winload writes to the disk during boot -- BCD updates, log entries, registry hive flushes. Without bounds checking, those writes corrupted adjacent hypervisor memory. This manifested as random crashes minutes into boot, which made it incredibly hard to diagnose.

Fix 5: MSR shadow

Related to Fix 2, but broader. The guest was reading MSRs that my hypervisor didn't handle -- not just MTRRs, but also IA32_MISC_ENABLE, various performance counter MSRs, and platform-specific MSRs that QEMU exposes. For any unhandled MSR read, I was returning zero. For some MSRs, zero is an invalid value that causes the guest to take a completely different code path (like disabling XD/NX, which then broke DEP).

The fix: comprehensive MSR shadowing. I capture the host's real MSR values at virtualization time and serve those as defaults for any MSR the guest reads that I don't specifically modify.

Fix 6: The 500 million exit limit

This one was embarrassing. I had a debug counter that incremented on every #VMEXIT. When it hit 500 million, I triggered a deliberate BSOD with a diagnostic code so I could inspect the state. I forgot to remove it. The Windows boot process takes hundreds of millions of exits -- between NVMe I/O, timer interrupts, CPUID calls, and MSR reads. My "debug safety net" was killing the boot at around the 70% mark of the loading screen.

Removing one if statement fixed the final crash. After six fixes applied over three days of debugging, Windows booted.