TLB_CONTROL: The Field Hardware Reads Fresh Every Time

AMD SVM has a vmcb_clean bits field that tells the CPU which VMCB sections haven't changed since the last VMRUN. If I mark a section as clean, the CPU skips reloading it from memory and uses its cached copy instead. Saves time on every exit. But TLB_CONTROL is not covered by any clean bit. The CPU reads it fresh from the VMCB on every single VMRUN, regardless of what I put in vmcb_clean.

I didn't know this for a while, and it caused a subtle ordering bug that led to stale TLB entries and random crashes.

How vmcb_clean works

The VMCB control area has a 32-bit field called vmcb_clean. Each bit corresponds to a group of VMCB fields. If I set a bit, I'm telling the CPU "I haven't modified these fields since the last VMRUN on this VMCB, so you can use your internal cache."

// vmcb_clean bit assignments:
const CLEAN_INTERCEPTS: u32 = 1 << 0;
const CLEAN_IOPM:       u32 = 1 << 1;
const CLEAN_ASID:       u32 = 1 << 2;
const CLEAN_TPR:        u32 = 1 << 3;
const CLEAN_NP:         u32 = 1 << 4;
// ... more bits ...

// After handling an exit, set clean bits for
// everything I didn't modify:
vmcb.control.vmcb_clean = CLEAN_INTERCEPTS
    | CLEAN_IOPM
    | CLEAN_ASID
    | CLEAN_NP;

This is a performance optimization. Without clean bits, the CPU reloads every VMCB field from memory on every VMRUN. With them, it skips the reload for unchanged fields, which shaves off a few hundred nanoseconds per exit. Over millions of exits, it adds up.

TLB_CONTROL is special

TLB_CONTROL sits at VMCB offset 0x058. It tells the CPU what to do with the TLB on the next VMRUN. The most common values are 0x00 (do nothing) and 0x01 (flush entire TLB). After an NPT page table modification, I set it to flush so the guest doesn't use stale translations.

// TLB_CONTROL values:
const TLB_DO_NOTHING:  u32 = 0x00;
const TLB_FLUSH_ALL:    u32 = 0x01;
const TLB_FLUSH_GUEST:  u32 = 0x03;
const TLB_FLUSH_NONGLOBAL: u32 = 0x07;

// In an NPF handler that modifies page tables:
vmcb.control.tlb_control = TLB_FLUSH_ALL;

// The CPU reads this FRESH on every VMRUN.
// No clean bit covers it.
// Whatever value is in memory at VMRUN time is what it uses.

The ordering bug

My exit handler had two separate places that wrote to TLB_CONTROL. First, the specific exit handler (like the NPF handler) would set TLB_FLUSH_ALL when it modified page tables. Then, the generic post-processing code at the end of the exit path would set TLB_DO_NOTHING as a reset for the next exit.

// The ordering problem:

// Step 1: NPF handler maps a new page
vmcb.control.tlb_control = TLB_FLUSH_ALL;

// Step 2: generic post-processing runs AFTER the handler
if action == ExitAction::Continue {
    vmcb.control.tlb_control = TLB_DO_NOTHING;
    vmcb.control.vmcb_clean = CLEAN_INTERCEPTS
        | CLEAN_IOPM | CLEAN_ASID | CLEAN_NP;
}

// VMRUN executes with TLB_DO_NOTHING
// The flush that the NPF handler requested never happens
// Guest uses stale TLB entry → accesses wrong physical page
// Random corruption, random crashes

If TLB_CONTROL were covered by a clean bit, the overwrite wouldn't matter -- the CPU would ignore it. But because the CPU always reads it fresh, the last write wins. And the last write was my generic reset code, which erased the handler's flush request.

The fix

The post-processing code now only sets TLB_DO_NOTHING if no handler has requested a flush during this exit. I use a flag in the per-CPU data to track whether any handler set a non-zero TLB_CONTROL value.

// Fixed post-processing:
if action == ExitAction::Continue {
    // only reset TLB_CONTROL if no handler set a flush
    if vmcb.control.tlb_control == TLB_DO_NOTHING {
        // nobody requested a flush, safe to leave it
    }
    // if a handler set TLB_FLUSH_ALL, leave it alone
    // it will be consumed by the next VMRUN
    // and we reset it on the NEXT exit's post-processing

    vmcb.control.vmcb_clean = CLEAN_INTERCEPTS
        | CLEAN_IOPM | CLEAN_ASID | CLEAN_NP;
}

Even simpler: I moved the TLB_DO_NOTHING reset to the beginning of the exit handler, before dispatching to specific handlers. The handler then overwrites it if needed. Since TLB_CONTROL is read at VMRUN time (which happens at the end), the handler's value is the one that takes effect.

The guest VMCB complication

This ordering bug got worse when I added the VMCB swap for guest VMs. The guest exit handler also needs to control TLB flushes -- when I modify guest NPT entries, I need to flush the guest's TLB before VMRUN'ing back into the guest. But the host's post-processing code was running on guest exits too, overwriting the guest's TLB_CONTROL.

The fix for the guest case was adding an is_guest_vm guard. Host post-processing skips guest VMCB exits entirely. The guest handler manages its own TLB_CONTROL. Two separate code paths, two separate VMCB configurations, no cross-contamination.

// Guard against host post-processing on guest exits:
if action == ExitAction::Continue {
    if !per_cpu.is_guest_vm {
        // host exit — apply host post-processing
        vmcb.control.vmcb_clean = CLEAN_INTERCEPTS
            | CLEAN_IOPM | CLEAN_ASID | CLEAN_NP;
    }
    // guest exits: handler already set TLB_CONTROL
    // don't touch it here
}