The MMIO Emulation Framework

After implementing TPM interception, HPET spoofing, and PM timer hiding, I noticed I was writing the same code three times. NPT unmap the page, catch the #NPF, decode the instruction, dispatch to a handler. Every MMIO device follows the same pattern. So I built a framework.

The idea is simple: register an address range and a callback. The framework handles all the ugly parts -- instruction decode, operand extraction, RIP advancement -- and your handler just gets a clean "this address was read, here's where to put the value" or "this address was written with this value." No more raw NPF exit info parsing in every handler.

Why MMIO needs interception

Hardware timers are the most common detection surface. The ACPI PM timer at 0xFED00000+0x608 ticks at 3.579545 MHz and can't be offset like the TSC. The HPET at 0xFED00000 provides nanosecond-resolution timestamps. Anti-cheats read these to cross-check against RDTSC -- if the TSC says 100 microseconds passed but the PM timer says 300, something is eating cycles in between. My hypervisor exits are that something.

The TPM at 0xFED40000 exposes firmware integrity state. Other MMIO devices -- IOAPIC, local APIC in xAPIC mode, PCI config space -- all need varying degrees of interception depending on what the guest is doing.

Each of these was its own one-off handler. The code was getting messy.

The dispatch table

I define a static array of MMIO region descriptors. Each one has a base address, a size, and a handler function pointer. On every #NPF where the faulting address falls in an MMIO range, the framework looks up the handler and calls it.

struct MmioRegion {
    base: PhysAddr,
    size: usize,
    handler: fn(u32, &mut u64, bool, u8),
}

static MMIO_REGIONS: [MmioRegion; 8] = [
    MmioRegion {
        base: PhysAddr(0xFED00608),
        size: 4,
        handler: handle_pm_timer,
    },
    MmioRegion {
        base: PhysAddr(0xFED00000),
        size: 0x400,
        handler: handle_hpet,
    },
    MmioRegion {
        base: PhysAddr(0xFED40000),
        size: 0x1000,
        handler: handle_tpm,
    },
    // ... IOAPIC, local APIC, etc.
];

Instruction decode

This is the ugly part. When the guest hits an unmapped MMIO page, I get an #NPF with the faulting GPA. But I also need to know: what instruction caused it? Was it a MOV read or write? What register is the source/destination? How wide is the access -- 1, 2, 4, or 8 bytes?

AMD SVM gives me the faulting RIP and the guest's CS base, so I can read the instruction bytes from guest memory. But x86 instruction encoding is a nightmare. Prefixes, REX bytes, ModRM, SIB, displacement -- every instruction has a different length and format. I need a mini-disassembler.

fn decode_mmio_instruction(
    rip: u64,
    cr3: u64,
) -> MmioAccess {
    let bytes = read_guest_bytes(cr3, rip, 15);
    let mut i = 0;
    let mut access_size = 4u8;
    let mut rex = 0u8;

    // skip prefixes
    while i < bytes.len() {
        match bytes[i] {
            0x66 => { access_size = 2; i += 1; }
            0x67 | 0xF0 | 0xF2 | 0xF3 |
            0x26 | 0x2E | 0x36 | 0x3E |
            0x64 | 0x65 => { i += 1; }
            0x40..=0x4F => {
                rex = bytes[i];
                if rex & 0x08 != 0 { access_size = 8; }
                i += 1;
            }
            _ => break,
        }
    }

    // decode MOV opcode + ModRM
    let opcode = bytes[i];
    let modrm = bytes[i + 1];
    let reg_idx = ((modrm >> 3) & 7) | ((rex & 4) << 1);

    let is_write = match opcode {
        0x89 | 0x88 => true,   // MOV r/m, r
        0x8B | 0x8A => false,  // MOV r, r/m
        _ => false,
    };

    MmioAccess { reg_idx, is_write, access_size,
        insn_len: compute_insn_len(bytes, i) }
}

I don't decode every possible x86 instruction. I only handle the MOV variants that compilers actually emit for MMIO accesses -- MOV r/m32, r32, MOV r32, r/m32, and their 8/16/64-bit variants via prefixes and REX. That covers readl()/writel() and their friends. If I encounter an instruction I don't recognize, I single-step it by temporarily mapping the page, setting the trap flag, and catching the #DB.

The NPF handler entry point

Everything comes together in the #NPF exit handler. When the CPU faults on a nested page, I check if the faulting GPA falls in any registered MMIO region. If it does, I decode the instruction, extract the operand, call the handler, write the result back to the guest register, and advance RIP.

fn handle_npf(
    svm: &mut SvmData,
    faulting_gpa: PhysAddr,
) {
    // check MMIO regions first
    for region in &MMIO_REGIONS {
        if region.contains(faulting_gpa) {
            let access = decode_mmio_instruction(
                svm.vmcb.save.rip,
                svm.vmcb.save.cr3,
            );
            let offset = (faulting_gpa.as_u64() - region.base.as_u64()) as u32;
            let reg = svm.guest_reg_mut(access.reg_idx);

            // call the device handler
            (region.handler)(offset, reg, access.is_write, access.access_size);

            // advance past the instruction
            svm.vmcb.save.rip += access.insn_len as u64;
            return;
        }
    }

    // not MMIO -- handle as normal NPF (page hiding, etc.)
    handle_regular_npf(svm, faulting_gpa);
}

Adding a new device

With the framework in place, adding a new MMIO intercept takes about 20 lines. I added IOAPIC interception in under an hour when I needed to spoof interrupt routing for the guest VM. No instruction decode code. No NPT manipulation. Just write a handler that takes an offset and returns a value.

// adding a new MMIO device -- just the handler function
fn handle_ioapic(
    offset: u32,
    val: &mut u64,
    is_write: bool,
    _size: u8,
) {
    match (offset, is_write) {
        (0x00, true)  => { IOREGSEL = *val as u32; }
        (0x10, false) => {
            // IOWIN read -- return spoofed or real value
            *val = read_ioapic_reg(IOREGSEL);
        }
        _ => { // pass through }
    }
}

The framework currently handles six devices: PM timer, HPET, TPM, IOAPIC, local APIC (xAPIC mode), and a region I reserve for PCI MMIO config space interception. Each handler is a pure function that takes an offset and a value reference. No global state, no NPT knowledge, no instruction decoding. All of that complexity is in the framework layer.

The single-step fallback

Sometimes the compiler emits something I don't decode -- a MOVZX, a MOVSX, a string operation, or something with a weird prefix combination. When that happens, I fall back to single-stepping: temporarily map the MMIO page in NPT, set the guest's RFLAGS.TF (trap flag), and resume. The CPU executes the one instruction against real hardware, then immediately traps on #DB. I catch the #DB, unmap the page again, clear TF, and continue.

This fallback fires maybe once per ten thousand MMIO accesses. It's slower -- two extra VMEXITs -- but it means I never fail to emulate an instruction. The framework handles everything, and the device handlers never need to know about it.

What this enables

Before the framework, every new MMIO intercept was a project. After it, adding TPM emulation was an afternoon. Adding PM timer compensation was 30 lines. The instruction decode is the hardest part of MMIO emulation, and it's written exactly once. Everything else is just "what value should this register return."

This is one of the pieces I'm most satisfied with architecturally. It's clean, it's fast, and it turned a class of problems into a table lookup.