MTRR Shadow: Memory Type Emulation

Memory Type Range Registers tell the CPU how to cache different regions of physical memory -- write-back, uncacheable, write-combining, and so on. The Windows kernel reads MTRRs during early boot to build its cache attribute tables. My hypervisor was returning whatever the host had, which made no sense for the guest's view of physical memory. I needed a shadow.

This was the second wall in the Windows boot sequence, right after the GDT fix. With segment registers sorted out, the kernel initialized KdInitSystem (debugger), then moved into MmInitSystem (memory manager). That's where it started reading MTRR MSRs and found values that didn't correspond to any valid memory layout. The memory manager panicked and the boot crashed.

What MTRRs actually do

The CPU has two mechanisms for controlling cache behavior: PAT (Page Attribute Table) and MTRRs (Memory Type Range Registers). PAT works at the page table level -- each PTE has bits that select a memory type. MTRRs work at the physical address level -- they define ranges of physical memory and assign each range a type. The final cache behavior is determined by combining both.

There are two kinds of MTRRs. Fixed-range MTRRs cover the first 1MB of physical memory in fine-grained 4KB/16KB/64KB chunks. Variable-range MTRRs cover arbitrary regions using a base/mask pair -- typically used for the main RAM range (write-back) and MMIO ranges (uncacheable).

// the MSRs the kernel reads during MmInitSystem
// fixed-range MTRRs (first 1MB)
MSR 0x250   // MTRR_FIX64K_00000
MSR 0x258   // MTRR_FIX16K_80000
MSR 0x259   // MTRR_FIX16K_A0000
MSR 0x268   // MTRR_FIX4K_C0000 (through 0x26F)

// variable-range MTRRs (arbitrary regions)
MSR 0x200   // MTRR_PHYSBASE0
MSR 0x201   // MTRR_PHYSMASK0
// ... pairs up to PHYSBASE7/PHYSMASK7 (0x20E/0x20F)

// default type + enable bits
MSR 0x2FF   // MTRR_DEF_TYPE

The problem: host MTRRs don't match guest reality

Before I added the shadow, my MSR intercept handler had a simple rule: if I don't specifically handle an MSR, pass the read through to real hardware. This worked fine for most MSRs. But MTRR MSRs describe the host's physical memory layout, not the guest's.

My host has 32GB of RAM. The MTRRs say physical addresses 0-32GB are write-back, and everything above that is uncacheable. But my guest only has 8GB of RAM mapped. The guest reads the host's MTRRs, sees write-back ranges extending to 32GB, tries to access memory at 16GB (which doesn't exist in the guest), and gets confused. The memory manager's MTRR parsing assumes the MTRR layout matches the actual physical memory it can see.

Building the MTRR shadow

The shadow is a software copy of MTRR state. When the guest reads an MTRR MSR, my handler returns the shadow value instead of the real hardware value. When the guest writes an MTRR MSR, I store the value in the shadow but don't touch real hardware. The guest thinks it's programming the MTRRs; in reality, it's talking to my data structure.

struct MtrrShadow {
    def_type: u64,
    var_base: [u64; 8],
    var_mask: [u64; 8],
    fix_64k: u64,
    fix_16k: [u64; 2],
    fix_4k: [u64; 8],
}

impl MtrrShadow {
    fn new(guest_ram_size: u64) -> Self {
        let mut s = Self::default();

        // enable MTRRs, enable fixed-range, default type = WB
        s.def_type = 0x0C06;  // bits 11,10 set + type 6 (WB)

        // fixed range: all WB for first 640KB, UC for VGA hole
        s.fix_64k = 0x0606060606060606;  // 8x WB
        s.fix_16k[0] = 0x0606060606060606;  // 80000-9FFFF: WB
        s.fix_16k[1] = 0x0000000000000000;  // A0000-BFFFF: UC (VGA)

        // variable range 0: main RAM as WB
        s.var_base[0] = 0x0000000006;   // base 0, type WB
        s.var_mask[0] = compute_mask(guest_ram_size) | (1 << 11);

        // variable range 1: MMIO region as UC
        s.var_base[1] = 0x00000C0000000000;  // base 0xC0000000, UC
        s.var_mask[1] = 0x0000FFC000000800;  // covers 1GB MMIO hole

        s
    }
}

I initialize the shadow with a clean layout that matches the guest's actual memory configuration. Guest RAM (8GB) is write-back. The MMIO hole at 0xC0000000 (where PCI BARs live) is uncacheable. The VGA frame buffer region at 0xA0000-0xBFFFF is uncacheable. Everything else follows the default type.

The MSR intercept handler

The handler is a big match statement on the MSR number. For MTRR reads, I return shadow values. For MTRR writes, I store into the shadow. Everything else passes through to hardware.

fn handle_msr_read(msr: u32, shadow: &MtrrShadow) -> u64 {
    match msr {
        0x2FF => shadow.def_type,
        0x250 => shadow.fix_64k,
        0x258 => shadow.fix_16k[0],
        0x259 => shadow.fix_16k[1],
        0x268..=0x26F => shadow.fix_4k[(msr - 0x268) as usize],
        0x200..=0x20F => {
            let idx = ((msr - 0x200) / 2) as usize;
            if msr % 2 == 0 { shadow.var_base[idx] }
            else { shadow.var_mask[idx] }
        }
        0xFE => 0x0000000000000D08,  // MTRRcap: 8 var, fix, WC
        _ => rdmsr(msr),  // everything else: real hardware
    }
}

fn handle_msr_write(msr: u32, val: u64, shadow: &mut MtrrShadow) {
    match msr {
        0x2FF => shadow.def_type = val,
        0x250 => shadow.fix_64k = val,
        // ... same pattern for all MTRR MSRs
        0x200..=0x20F => {
            let idx = ((msr - 0x200) / 2) as usize;
            if msr % 2 == 0 { shadow.var_base[idx] = val; }
            else { shadow.var_mask[idx] = val; }
        }
        _ => wrmsr(msr, val),  // non-MTRR: write to real hardware
    }
}

The stealth angle

MTRR shadowing isn't just about boot correctness -- it's also a stealth concern. If a detection tool reads MTRR MSRs and compares them against the actual detected memory layout, inconsistencies reveal the hypervisor. The host might have 32GB of RAM with MTRRs covering that full range, but the guest only sees 8GB. If I pass through the host's MTRRs, a tool that cross-references e820 memory map entries against MTRR ranges would notice the mismatch.

The shadow ensures the guest sees MTRR values that exactly match its own physical memory layout. No detection tool can find an inconsistency because there isn't one -- the shadow is internally consistent with the guest's view of the world.

What I learned

MTRRs are one of those things you never think about until they break. On bare metal, the BIOS sets them up during POST and the OS trusts whatever it finds. In a hypervisor, you have to actively manage them because the guest's physical memory is not the host's physical memory. The boot path reads them early, the memory manager depends on them for cache correctness, and detection tools can use them to fingerprint the environment.

After this fix, MmInitSystem completed successfully. The kernel moved on to loading boot-start drivers, which is where the next problem -- incomplete disk I/O -- was waiting. But the MTRR shadow stayed solid. I never had to touch it again after the initial implementation. That's the best kind of fix.