GPU VRAM Overlay: Hiding in Video Memory

CPU-based memory scanners can walk every page of physical RAM. They can dump kernel pools, scan non-paged allocations, checksum driver images. But they cannot see GPU VRAM. The framebuffer lives on the discrete GPU's own memory -- separate address space, separate bus, completely invisible to MmCopyMemory or any physical page walk. I wanted to put hypervisor data there.

The idea is simple: write structured data into VRAM at known offsets, then read it back when needed. The CPU can access VRAM through the GPU's BAR (Base Address Register) -- a PCI MMIO window. On NVIDIA, that's BAR1 for the primary framebuffer and BAR0+0x700000 (PRAMIN) for a configurable 1MB window into arbitrary VRAM offsets.

The problem is that GPUs don't store pixels linearly. They use tiled memory layouts with swizzle patterns that rearrange bytes for cache locality during rendering. If I write a linear byte stream to VRAM, it comes out scrambled on screen -- but more importantly, reading it back requires knowing the exact tiling formula to unscramble it.

Why tiling exists

When a GPU rasterizes a triangle, it doesn't draw one scanline at a time like a CPU blitter. It processes pixels in 2D blocks -- typically 8x8 or 16x16 tiles. The memory layout is optimized so that spatially adjacent pixels are also adjacent in memory, which means better cache hit rates during texture sampling and fragment shading.

For the CPU, this is a nightmare. A "linear" write to VRAM offset 0x1000 doesn't correspond to pixel (X, Y) at the position I'd expect. The GPU's memory controller applies a swizzle function that XORs bit positions to create a space-filling curve through the tile. Different GPU architectures use different swizzle patterns.

The Ampere/Turing swizzle formula

I spent two days trying to reverse the tiling formula from NVIDIA's proprietary driver before finding it in the Linux kernel. The nouveau DRM panic handler (merged in Linux 6.13) has the exact formula for rendering text directly to VRAM during kernel panics -- which means it needs to convert pixel coordinates to VRAM byte offsets without any GPU acceleration.

// Ampere/Turing XOR swizzle -- from Linux nouveau drm_panic
// tile_shift=4 for DWM composition surfaces at 1080p
const BLOCK_HEIGHT: u32 = 128;
const BLOCK_W: u32 = 16;

fn pixel_to_vram_offset(x: u32, y: u32, pitch: u32) -> usize {
    let bpp = 4; // 32-bit BGRA
    let blk_cols = pitch / (BLOCK_W * bpp);
    let bx = x / BLOCK_W;
    let by = y / BLOCK_HEIGHT;
    let blk_off = (by * blk_cols + bx) * 64 * BLOCK_HEIGHT;

    let lx = x % BLOCK_W;  // local X within 16px block
    let ly = y % BLOCK_HEIGHT;
    let swz = (lx & 3)
        | ((ly & 3) << 2)
        | ((lx & 4) << 2)
        | ((ly & 4) << 3)
        | ((lx & 8) << 3)
        | ((ly >> 3) << 7);

    (blk_off + swz * bpp) as usize
}

The key insight is that this operates in pixel coordinates within 16-pixel-wide blocks, not byte coordinates within 64-byte GOBs. I originally had a homegrown formula that worked in byte coordinates and it produced garbage -- pixels scattered across the screen instead of forming a rectangle. The Linux kernel formula just works.

Finding DWM composition surfaces

The Desktop Window Manager triple-buffers its composition output. There are typically 3 active surfaces among the first 6 allocated in VRAM. I scan the first 512MB of VRAM through PRAMIN and identify surfaces by checking for valid pixel data.

// Surface detection via PRAMIN (BAR0 + 0x700000)
fn scan_vram_surfaces(bar0: *mut u8) -> Vec<VramSurface> {
    let pramin = bar0.add(0x700000);
    let mut surfaces = Vec::new();

    for mb in 0..512 {
        set_pramin_target(bar0, mb * 0x100000);
        let first_pixel = read_u32(pramin);
        // valid surface: alpha channel = 0xFF at offset 0
        if (first_pixel >> 24) != 0xFF { continue; }
        // span check: also valid at MB+7 offset
        set_pramin_target(bar0, (mb + 7) * 0x100000);
        if (read_u32(pramin) >> 24) != 0xFF { continue; }
        surfaces.push(VramSurface { offset: mb * 0x100000 });
        if surfaces.len() >= 6 { break; }
    }
    surfaces
}

I dedup surfaces with a 2MB gap to avoid counting the same allocation twice. DWM's triple buffering means 3 of these 6 are actively being composited. I write to all of them to ensure the overlay is always visible regardless of which buffer is currently displayed.

What this actually looks like

With the correct swizzle formula and surface detection, I can draw a 100x100 pixel rectangle at arbitrary screen coordinates, directly in VRAM, at roughly 50fps across all 6 surfaces. The rectangle appears on the physical display but is completely invisible to any DXGI-based capture -- PrintScreen, Win+Shift+S, OBS, any screen recording software. They all capture from the GPU's output pipeline, not from raw VRAM bytes.

There is minor flicker when typing in cmd because DWM recomposes the desktop and briefly shows a frame where my overlay hasn't been rewritten yet. But when the desktop is idle, it's rock solid.

Limitations and future work

The tile_shift parameter varies per surface. I've confirmed tile_shift=4 for DWM desktop composition at 1080p, but different resolutions or surface formats might use different values. For AMD GPUs, the tiling formula is completely different -- they use a macro-tile + micro-tile system with bank interleaving that I haven't reversed yet.

The PRAMIN window is also a bottleneck. It's only 1MB, so writing to different VRAM regions requires constantly repointing it. A faster path would be to map a larger VRAM region through BAR1, but that requires understanding NVIDIA's page table engine (the GPU has its own MMU) to set up the mapping.

For now, this is a working proof of concept. Hypervisor data in VRAM, invisible to CPU scanners, readable back through the same BAR path. The swizzle formula was the hard part -- everything else is just PCI MMIO reads and writes.