Features I Stripped for Stability

During stability testing, I stripped five features that each worked in isolation but caused intermittent crashes under load. Every one of them was something I spent days building. Every one of them had to go. The process was always the same: add the feature, stress test for an hour, hit a BSOD that only reproduces once in a hundred boots, strip the feature, move on.

This isn't a failure post. It's about prioritization. The features went into a backlog, not the trash. I can add them back once the foundation is solid. But a hypervisor that crashes once every four hours is useless regardless of how many features it has.

What was stripped

I started with the full exit handler -- every optimization I'd built over months. Then I cut it down to the bare minimum that could virtualize and devirtualize cleanly. Here's what went away.

// Original handle_exit — full featured:
pub unsafe fn handle_exit(
    per_cpu: &mut PerCpu,
    regs: &mut GuestRegs
) -> ExitAction {
    per_cpu.vmexit_count += 1;

    if per_cpu.devirt_requested {
        // devirtualize — this stays
        return ExitAction::Devirtualize;
    }

    let enter_tsc = timing::exit_enter();
    let action = dispatch_exit(per_cpu, regs);

    if action == ExitAction::Continue {
        vmcb.control.tlb_control = TLB_DO_NOTHING;
        vmcb.control.vmcb_clean = CLEAN_INTERCEPTS
            | CLEAN_IOPM | CLEAN_ASID | CLEAN_NP;
        timing::exit_leave(per_cpu, enter_tsc);
    }
    action
}

1. VMEXIT count tracking

Simple u64 increment on every exit. Why strip it? Because the per-CPU data struct was being accessed at GIF=0 (global interrupt flag disabled), and the counter increment touches a cache line that other cores might also be reading if I have shared diagnostic data nearby. Under heavy load with all 16 cores exiting simultaneously, this created cache line bouncing. Not the root cause of the BSOD, but it added enough latency variance to push other timing-sensitive code over the edge.

I moved it to a separate cache-line-aligned field later and added it back. But during the "find the crash" phase, it had to go.

2. TSC timing compensation

The timing::exit_enter() and timing::exit_leave() calls bracket the handler to measure how many cycles the exit took, then subtract that from the guest's TSC offset. This is the core of my stealth timing -- without it, the guest can trivially detect the hypervisor by measuring RDTSC gaps around intercepted instructions.

// TSC compensation — measures and subtracts handler cost
fn exit_enter() -> u64 {
    core::arch::x86_64::_rdtsc()
}

fn exit_leave(per_cpu: &mut PerCpu, enter: u64) {
    let cost = core::arch::x86_64::_rdtsc() - enter;
    let vmcb = &mut *per_cpu.vmcb;
    vmcb.control.tsc_offset -= cost as i64;
}

Stripped it because the RDTSC calls themselves were adding variability to the exit path. The compensation was slightly over- or under-correcting depending on cache state, and I needed a clean baseline to find the actual crash. Stealth is important, but a stable hypervisor that's detectable is more useful than an undetectable one that BSODs.

3. Trace buffer

A ring buffer that records the last N exits -- exit code, RIP, info1, info2, RCX. Invaluable for post-mortem debugging. But the ring buffer write on every exit was the most expensive operation in the fast path. Each entry is 40 bytes, and at millions of exits per second, the buffer cycles through L1 cache constantly.

// Trace buffer — ring buffer of recent exits
const TRACE_BUF_SIZE: usize = 1024;

struct ExitTrace {
    exit_code: u64,
    rip: u64,
    info1: u64,
    info2: u64,
    rcx: u64,
}

// In dispatch_exit:
let idx = per_cpu.trace_idx % TRACE_BUF_SIZE;
per_cpu.trace_buf[idx] = ExitTrace {
    exit_code: ec, rip: vmcb.save.rip,
    info1: vmcb.control.exit_info1,
    info2: vmcb.control.exit_info2,
    rcx: regs.rcx,
};
per_cpu.trace_idx = per_cpu.trace_idx.wrapping_add(1);

I keep the trace buffer code ready to re-enable at any time. When I'm debugging, I turn it back on. When I'm testing stability, it goes.

4. TLB_CONTROL reset and 5. VMCB clean bits

These are performance optimizations that tell the CPU to cache VMCB state between exits. They're supposed to make things faster. But when I was chasing a crash, I needed to eliminate every variable. If the CPU's internal VMCB cache was somehow stale (unlikely but not impossible), these bits would be hiding it.

Stripping them forces the CPU to reload the entire VMCB from memory on every VMRUN. Slower, but deterministic. If the crash disappears without clean bits, it points at a caching issue. If the crash persists, the clean bits aren't the problem and I can add them back.

The stripped-down handler

After stripping everything, the exit handler was about 15 lines of Rust. Check for devirt request, dispatch to the specific handler, return. No counting, no timing, no tracing, no caching. Just the bare minimum to keep Windows running as a guest.

And that bare minimum was stable. Hours of stress testing, no BSOD. The features I stripped weren't the root cause, but removing them reduced the noise enough to find it. Once I fixed the interrupt stack issue, I added everything back and it's been stable since.

The features became a backlog, not lost work. Every one of them is back in the codebase now. But the lesson stuck: stability first, features second. A hypervisor that runs for days without crashing is worth more than one with perfect TSC compensation that BSODs every four hours.