Atomic SIPI Delivery: Waking Up Application Processors

In stage 9, the guest VM boots OVMF on two physical cores. The BSP (Bootstrap Processor) starts executing firmware, discovers there's a second core in the ACPI tables, and sends a Startup IPI to wake it up. The problem: the BSP can't directly touch the AP's VMCB. Hardware owns that structure. Modifying another core's active VMCB is a race against the CPU itself.

This was the hardest multicore synchronization problem I hit in the entire project. The solution is borrowed from KVM: atomic flags, not direct VMCB modification.

How SIPI works on bare metal

On real hardware, the BSP sends an INIT IPI followed by a Startup IPI (SIPI) to the AP. The SIPI carries an 8-bit vector in the ICR. The AP wakes up in real mode at address vector << 12 -- so vector 0x10 means the AP starts executing at physical address 0x10000. CS is set to vector << 8, IP is 0.

In a hypervisor, the guest writes to the ICR to send an IPI. I intercept that write. But the AP is currently sitting in a HLT exit loop on a different physical core, inside its own VMRUN. I can't reach into its VMCB and change CS:RIP -- the CPU will overwrite my changes on the next #VMEXIT because it saves the guest state into the VMCB save area on every exit.

// this is what you might try -- and it DOESN'T WORK

// BSP exit handler, running on core 14:
let ap_vmcb = get_ap_vmcb(target_apic_id);

// set AP to real mode at vector << 12
ap_vmcb.save.cs.selector = (vector as u16) << 8;
ap_vmcb.save.rip = 0;

// this races with core 15's VMRUN/VMEXIT cycle
// the AP will never see these values

The atomic flag pattern

The BSP doesn't modify the AP's VMCB. Instead, it sets two atomic variables: a boolean "SIPI pending" flag and a u8 holding the SIPI vector. The AP, sitting in its HLT exit handler, checks these flags on every exit loop iteration.

// per-AP state -- shared between cores via atomic operations
struct ApSipiState {
    sipi_pending: AtomicBool,
    sipi_vector:  AtomicU8,
    init_received: AtomicBool,
}

// BSP side -- on core 14, handling guest ICR write
fn handle_ipi_from_guest(
    icr_value: u64,
    dest: u32,
) {
    let delivery_mode = (icr_value >> 8) & 0x7;
    let vector = (icr_value & 0xFF) as u8;
    let phys_target = virtual_to_phys_apic_id(dest);
    let ap_state = &AP_SIPI_STATES[phys_target as usize];

    match delivery_mode {
        0x5 => {
            // INIT IPI
            ap_state.init_received.store(true, Ordering::Release);
        }
        0x6 => {
            // Startup IPI
            if ap_state.init_received.load(Ordering::Acquire) {
                ap_state.sipi_vector.store(vector, Ordering::Release);
                ap_state.sipi_pending.store(true, Ordering::Release);
            }
        }
        _ => { // other IPI types -- forward to real hardware }
    }
}

The AP side -- HLT exit handler

The AP starts in a HLT loop. The guest firmware puts unused APs in HLT, which triggers a VMEXIT. In my exit handler, I check for a pending SIPI. When I see one, I do the full real-mode reset on my own VMCB -- no cross-core writes, no races.

fn handle_hlt_exit(svm: &mut SvmData) {
    let my_core = current_core_id();
    let state = &AP_SIPI_STATES[my_core];

    if state.sipi_pending.load(Ordering::Acquire) {
        let vector = state.sipi_vector.load(Ordering::Acquire);

        // full real-mode reset on MY OWN VMCB
        svm.vmcb.save.cs.selector = (vector as u16) << 8;
        svm.vmcb.save.cs.base = (vector as u64) << 12;
        svm.vmcb.save.cs.limit = 0xFFFF;
        svm.vmcb.save.cs.attrib = 0x009B; // 16-bit code, RW, accessed

        // RIP = 0 -- start at base of CS segment
        svm.vmcb.save.rip = 0;

        // data segments: real-mode defaults
        reset_data_segments(&mut svm.vmcb.save);

        // CR0: real mode (no PE, no PG)
        svm.vmcb.save.cr0 = 0x00000010; // ET bit only
        svm.vmcb.save.rflags = 0x02;     // reserved bit

        // clear the flag
        state.sipi_pending.store(false, Ordering::Release);
        state.init_received.store(false, Ordering::Release);

        // don't re-enter HLT -- fall through to VMRUN
        // the AP will start executing guest real-mode code
        return;
    }

    // no SIPI pending -- re-execute HLT
    // (don't advance RIP, so VMRUN re-hits the HLT)
}

The GIF=0 spin-wait

There's a subtlety here. The AP sits in its HLT exit handler, which runs at GIF=0 (Global Interrupt Flag cleared by hardware on VMEXIT). No interrupts can fire. The BSP sets the atomic flag from its own exit handler, also at GIF=0. Both cores are in their exit handlers simultaneously, communicating through atomics.

The AP doesn't busy-spin though. After checking for SIPI and finding none, it just does VMRUN again, which re-executes the guest's HLT instruction. The next VMEXIT from HLT comes almost immediately -- HLT exits are fast because the guest isn't doing any useful work. So the effective polling rate is "every HLT exit cycle," which is sub-microsecond.

// the AP's exit loop -- simplified
loop {
    // VMRUN returns here on every #VMEXIT
    let exit_code = svm.vmcb.control.exit_code;

    match exit_code {
        VMEXIT_HLT => {
            handle_hlt_exit(svm);
            // if SIPI was delivered, next VMRUN starts real-mode code
            // if not, next VMRUN re-executes HLT, exits again
        }
        VMEXIT_CPUID => handle_cpuid(svm),
        VMEXIT_MSR => handle_msr(svm),
        // ... other exit handlers
        _ => {}
    }

    // VMRUN -- guest resumes (or re-halts)
    vmrun(svm);
}

Why not just use a real IPI?

I could let the BSP's IPI hit real hardware -- send a physical INIT+SIPI to core 15. But that would reset the AP's CPU state outside of my hypervisor's control. The physical core would jump to the SIPI vector in real mode, but it's not in a VMRUN loop anymore -- it's bare metal. I'd have to re-virtualize it, which means another VMCB setup, another VMRUN, and a complex state machine to handle the transition. The atomic flag approach keeps the AP inside VMRUN the entire time. It never leaves guest mode. Much simpler.

The result

OVMF boots, discovers 2 CPUs in the MADT, sends INIT+SIPI to APIC ID 1 (which I translate to core 15). My BSP handler sets the atomic flags. Core 15's HLT handler sees the SIPI, resets to real mode at the firmware's AP trampoline address, and starts executing 16-bit startup code. Within milliseconds, both vCPUs are in 64-bit long mode running UEFI firmware.

The whole sequence -- INIT IPI, wait 10ms, SIPI, AP wake, AP enters protected mode, AP enters long mode -- takes about 50ms. Most of that is the firmware's own delays. My hypervisor overhead is negligible.