GuestRegs vs GuestGprs: The Layout Bug That Froze Everything

I had two register save structs with the same fields in opposite order. One saves r15 first, the other saves rcx first. When a handler casts a pointer to the wrong struct type, every single register read is wrong. RCX gets R15's value. RAX gets R14's value. The handler silently corrupts everything it touches, and the system freezes.

This is the kind of bug that doesn't crash immediately. It corrupts state quietly, and the actual failure happens hundreds of exits later in a completely unrelated code path. I spent a full day before I realized I was looking at the wrong struct.

Two structs, opposite order

The blue-pill hypervisor (stage 7) has an assembly stub that saves general-purpose registers on every #VMEXIT. It pushes them in a specific order -- r15 first, down to rcx. The Rust struct that maps over this memory is called GuestRegs, and its fields match that push order.

// Blue-pill asm (vmrun.rs) — pushes r15 FIRST
// Stack grows down, so r15 is at the lowest address
push r15
push r14
push r13
push r12
push r11
push r10
push r9
push r8
push rdi
push rsi
push rbp
push rbx
push rdx
push rcx

// GuestRegs struct layout (matches push order):
struct GuestRegs {
    r15: u64,  // offset 0x00
    r14: u64,  // offset 0x08
    r13: u64,  // offset 0x10
    // ... down to ...
    rdx: u64,  // offset 0x60
    rcx: u64,  // offset 0x68
}

Then I built the guest VM system (stage 8/9). The guest's VMRUN stub pushes registers in the opposite order -- rcx first, up to r15. Its struct is called GuestGprs.

// Guest asm (vmrun_test.rs) — pushes rcx FIRST
push rcx
push rdx
push rbx
push rbp
push rsi
push rdi
push r8
push r9
push r10
push r11
push r12
push r13
push r14
push r15

// GuestGprs struct layout (matches this push order):
struct GuestGprs {
    rcx: u64,  // offset 0x00
    rdx: u64,  // offset 0x08
    // ... up to ...
    r14: u64,  // offset 0x60
    r15: u64,  // offset 0x68
}

The cast that breaks everything

Both structs are 14 fields of u64. Same size. Same alignment. A pointer cast from one to the other compiles without any warning. Rust doesn't care -- it's a raw pointer cast in unsafe code, and the memory is the right size.

// This compiles fine and looks reasonable:
let gprs = unsafe {
    &mut *(regs as *mut GuestRegs as *mut GuestGprs)
};

// Now every field access is wrong:
gprs.rcx  // ← actually reads GuestRegs.r15 (offset 0x00)
gprs.rdx  // ← actually reads GuestRegs.r14 (offset 0x08)
gprs.r15  // ← actually reads GuestRegs.rcx (offset 0x68)

// An NPF handler that reads gprs.rcx to get the
// faulting address gets r15 instead.
// It "handles" the fault for a completely wrong address.
// Guest resumes with corrupted state.

Why it was hard to find

With the wrong struct cast, the guest got garbage register values on every VMEXIT. OVMF would triple-fault almost immediately and the guest would shut down. And here's the cruel irony -- when the guest shuts down quickly, the swap storm doesn't happen. The system appeared stable because the guest died too fast to cause problems.

When I fixed the register layout (thinking I was fixing the real bug), the guest suddenly lived long enough to trigger the swap storm, and the system froze. I thought I'd introduced a bug by fixing the layout. It took me another several hours to realize I'd fixed one bug and uncovered a second one hiding behind it.

// The fix: NEVER cast between GuestRegs and GuestGprs.
// Build a local GuestGprs by copying fields by name:

let local_gprs = GuestGprs {
    rcx: regs.rcx,
    rdx: regs.rdx,
    rbx: regs.rbx,
    rbp: regs.rbp,
    rsi: regs.rsi,
    rdi: regs.rdi,
    r8:  regs.r8,
    r9:  regs.r9,
    r10: regs.r10,
    r11: regs.r11,
    r12: regs.r12,
    r13: regs.r13,
    r14: regs.r14,
    r15: regs.r15,
};
// After handler, copy modified fields back the same way

The rule

Public VMEXIT handlers MUST use GuestRegs fields directly. Never cast to GuestGprs. If a handler needs the GuestGprs layout for some reason, build a local copy by field name and copy back when done. The two structs happen to have the same size, but their layouts are mirrors of each other.

I should have caught this earlier. The two structs existed because I wrote the guest assembly months after the blue-pill assembly, and I used the more natural rcx, rdx, ... push order for the guest. In retrospect, I should have used the same push order everywhere. Two different layouts for the same logical data is a bug factory.