The 500 Million Exit Limit

I had a safety limit in my hypervisor: if the exit counter exceeded 500 million, kill the guest. It was a development guard against infinite loops. Windows boot hit it legitimately. My own safety code was the bug.

This is a short post because the fix is trivial -- change a constant. But the lesson isn't. I had spent days debugging the GDT, the MTRRs, and the disk I/O. Everything was working. The boot was progressing further than ever before. Drivers were loading. Then the guest just... stopped. No crash, no BSOD. The serial output went silent and my hypervisor log said "exit limit reached."

Why the limit existed

When I first started building the guest VM infrastructure, I was terrified of infinite loops. A bug in the #VMEXIT handler could cause the same exit to fire repeatedly -- the handler fails to advance RIP, so the guest re-executes the same instruction, triggers the same exit, hits the same handler, forever. The machine appears frozen because the core is trapped in a VMRUN/#VMEXIT loop that never progresses.

To catch this during development, I added a counter:

static mut EXIT_COUNT: u64 = 0;

const MAX_EXITS: u64 = 500_000_000;

fn vmexit_handler(vmcb: &mut VMCB) -> VmAction {
    unsafe { EXIT_COUNT += 1; }

    if unsafe { EXIT_COUNT } > MAX_EXITS {
        log!("EXIT LIMIT: {} exits, killing guest", unsafe { EXIT_COUNT });
        return VmAction::Kill;
    }

    // ... handle the actual exit
}

500 million felt like an absurdly high number. During OVMF testing, the guest would boot to the UEFI shell in about 2-3 million exits. Even stress-testing OVMF with multiple reboots, I never exceeded 20 million. A threshold of 500 million was 250x headroom. Surely nothing would ever hit it.

Windows boot: a different scale

I added per-exit-type counters to understand the breakdown. Windows boot is relentless:

// exit type breakdown from a full Windows boot
MSR access:      412,000,000  // 51.5% of all exits
I/O port:        198,000,000  // 24.8%
CPUID:           156,000,000  // 19.5%
NPF (MMIO):       31,000,000  // 3.9%
other:            3,000,000  // 0.4%
// total:         ~800,000,000 exits

800 million exits. MSR accesses alone account for more than my entire safety limit. Windows reads MSRs constantly -- IA32_TSC_AUX on every context switch, IA32_SPEC_CTRL on every kernel entry (Spectre mitigations), IA32_KERNEL_GS_BASE on every syscall. Each one is an exit because my MSR bitmap intercepts them.

CPUID is the second biggest category. Every driver that loads calls CPUID to check CPU feature flags. The NVMe driver does it. The network driver does it. win32k.sys does it. The .NET runtime does it. 156 million CPUID calls during a single boot.

The fix

I raised the limit and added a disable flag:

// the fix: 10 billion limit + configurable disable
const MAX_EXITS: u64 = 10_000_000_000;
static LIMIT_ENABLED: AtomicBool = AtomicBool::new(true);

fn vmexit_handler(vmcb: &mut VMCB) -> VmAction {
    unsafe { EXIT_COUNT += 1; }

    if LIMIT_ENABLED.load(Ordering::Relaxed) {
        if unsafe { EXIT_COUNT } > MAX_EXITS {
            log!("EXIT LIMIT: {}", unsafe { EXIT_COUNT });
            return VmAction::Kill;
        }
    }

    // milestones -- log progress
    if unsafe { EXIT_COUNT } % 100_000_000 == 0 {
        log!("exit milestone: {}M", unsafe { EXIT_COUNT } / 1_000_000);
    }

    // ... handle the actual exit
}

I also added milestone logging every 100 million exits. This gives a rough progress indicator during boot without flooding the log. Watching the milestones tick up -- 100M, 200M, 300M -- gave me a sense of how far along the boot was. It became a habit: if milestones stop incrementing, something is stuck. If they're advancing steadily, the boot is making progress.

The real lesson

Don't put arbitrary safety limits on things when you don't know the scale of the real workload. My limit was calibrated against OVMF -- a minimal UEFI environment that boots in seconds and barely touches the hardware. Windows is three orders of magnitude more demanding. 3 million exits versus 800 million.

If I'd done the math upfront -- each CPUID takes an exit, each MSR access takes an exit, the average Windows driver calls CPUID 50-100 times during initialization, there are hundreds of drivers -- I would have predicted something in the hundreds of millions range. But I didn't do the math. I picked a number that "felt big" and moved on. Classic development shortcut that turned into a debugging session.

The numbers

For reference, here's the exit count comparison across the workloads I've tested:

// exit counts by workload
OVMF shell:          2,800,000
Linux (minimal):     47,000,000
Windows 10 boot:    800,000,000
Windows (tuned):     340,000,000

The 2.3x reduction from MSR bitmap tuning is significant. Each exit costs about 150 cycles of overhead. 460 million fewer exits at 150 cycles each is roughly 69 billion cycles saved -- about 19 seconds of CPU time on my 3.6GHz Ryzen. That's a measurable chunk of the boot time.

MSR bitmap tuning

The MSR permission map (MSRPM) is a 8KB bitmap that tells the CPU which MSR accesses to intercept. By default, I had everything intercepted -- every RDMSR and WRMSR caused a #VMEXIT. That's the safe but slow approach. After seeing that MSR exits dominated the exit count, I went through every MSR that Windows reads during boot and classified them.

MSRs that I need to intercept: anything related to Hyper-V synthetic MSRs (0x40000000-0x400000FF), MTRR MSRs (for the shadow), EFER (to keep SVME set), and STAR/LSTAR/CSTAR (for syscall monitoring). Everything else -- TSC_AUX, SPEC_CTRL, PRED_CMD, PERF_CTL, performance counters, temperature sensors -- can pass through to hardware without interception.

The bitmap change was a single function call per MSR: set the bit to 0 (passthrough) instead of 1 (intercept). The effect was dramatic. Boot time dropped from 45 seconds to about 28 seconds, and the machine felt noticeably more responsive during early startup.