Virtual APIC ID Spoofing

My guest VM runs on physical cores 14 and 15, but the guest OS needs to see APIC IDs 0 and 1. If it sees 14 and 15, the ACPI tables don't match, the MP table is wrong, and the OS either panics or refuses to bring up the second core. I had to lie about who the CPUs are.

This sounds like a small problem. It's not. APIC IDs leak through at least four different surfaces: CPUID, xAPIC MMIO registers, x2APIC MSRs, and ACPI tables. I have to intercept all of them consistently or the guest catches the mismatch.

Where APIC IDs come from

The initial APIC ID is baked into the CPU by the hardware. On my Ryzen 7 5800XT, core 14 has APIC ID 14, core 15 has APIC ID 15. The OS discovers these through multiple paths, and every path must agree.

// surfaces that expose the APIC ID

// 1. CPUID leaf 1, EBX[31:24] -- initial APIC ID
CPUID(1).EBX >> 24  // returns 14 on core 14

// 2. CPUID leaf 0Bh -- x2APIC/topology enumeration
CPUID(0x0B).EDX     // returns x2APIC ID

// 3. xAPIC MMIO -- local APIC register at 0xFEE00020
MMIO[0xFEE00020]   // APIC ID register

// 4. x2APIC MSR 0x802 -- IA32_X2APIC_APICID
RDMSR(0x802)        // x2APIC ID in EAX

If I fake CPUID but forget the xAPIC MMIO register, any code that reads the APIC ID through MMIO sees the real value. The OS checks these at boot, and if they disagree, bad things happen -- ranging from "wrong CPU affinity masks" to "kernel panic on topology validation."

The mapping table

I define a simple physical-to-virtual APIC ID mapping. Guest vCPU 0 runs on physical core 14, guest vCPU 1 runs on physical core 15. The mapping is stored per-core in my hypervisor data.

// GUEST_CORE_BASE = 14
// vCPU N runs on physical core (GUEST_CORE_BASE + N)

fn phys_to_virtual_apic_id(phys_apic_id: u32) -> u32 {
    if phys_apic_id >= GUEST_CORE_BASE {
        // map 14 -> 0, 15 -> 1
        phys_apic_id - GUEST_CORE_BASE
    } else {
        // not a guest core -- shouldn't happen
        phys_apic_id
    }
}

// used when guest writes APIC ID (e.g., IPI destination)
fn virtual_to_phys_apic_id(virt_apic_id: u32) -> u32 {
    virt_apic_id + GUEST_CORE_BASE
}

CPUID interception

CPUID is the easiest surface. I already intercept every CPUID in my blue-pill handler. I just added two more cases for the APIC ID leaves.

fn handle_cpuid_apic(svm: &mut SvmData) {
    let leaf = svm.guest_regs.rax as u32;

    match leaf {
        0x01 => {
            // execute real CPUID first
            native_cpuid(svm);
            // patch EBX[31:24] with virtual APIC ID
            let phys_id = (svm.guest_regs.rbx >> 24) as u32;
            let virt_id = phys_to_virtual_apic_id(phys_id);
            svm.guest_regs.rbx = (svm.guest_regs.rbx & 0x00FFFFFF)
                | ((virt_id as u64) << 24);
        }
        0x0B | 0x1F => {
            native_cpuid(svm);
            let phys_id = svm.guest_regs.rdx as u32;
            svm.guest_regs.rdx = phys_to_virtual_apic_id(phys_id) as u64;
        }
        _ => {}
    }
}

xAPIC MMIO interception

The local APIC is memory-mapped at 0xFEE00000. The APIC ID register is at offset 0x20. This is already handled by my MMIO framework -- I just register a handler for the APIC region.

The tricky part is that the guest also writes to the APIC for inter-processor interrupts (IPIs). When the guest's BSP sends an IPI to virtual APIC ID 1, I need to translate that to physical APIC ID 15 before the write hits hardware. It's a bidirectional translation.

fn handle_local_apic(
    offset: u32,
    val: &mut u64,
    is_write: bool,
    _size: u8,
) {
    match (offset, is_write) {
        (0x20, false) => {
            // APIC ID register read -- return virtual ID
            let phys_id = read_real_apic(0x20) >> 24;
            *val = (phys_to_virtual_apic_id(phys_id) as u64) << 24;
        }
        (0x310, true) => {
            // ICR high -- IPI destination field
            let dest = (*val >> 24) as u32;
            let phys_dest = virtual_to_phys_apic_id(dest);
            *val = (*val & 0x00FFFFFF) | ((phys_dest as u64) << 24);
            write_real_apic(0x310, *val as u32);
        }
        _ => {
            // everything else passes through
            if is_write { write_real_apic(offset, *val as u32); }
            else { *val = read_real_apic(offset) as u64; }
        }
    }
}

x2APIC MSR interception

If the guest enables x2APIC mode, the APIC registers move from MMIO to MSR space. MSR 0x802 is the APIC ID. MSR 0x830 is the ICR. I already have an MSR intercept bitmap -- I just mark these MSRs as intercepted and handle them in my RDMSR/WRMSR exit handler.

The result

With all four surfaces covered, the guest sees a perfectly consistent topology. CPUID says APIC ID 0 on the BSP and 1 on the AP. The APIC register agrees. x2APIC MSR agrees. The ACPI MADT table I provide lists APIC IDs 0 and 1. The guest boots both cores, assigns them to the right NUMA node, and never suspects they're actually cores 14 and 15.

OVMF boots to the BdsDxe menu with both vCPUs active. The MP table shows 2 processors. Everything just works -- as long as every surface tells the same lie.