kayfabe user manual

Kayfabe shares an NVIDIA GPU with virtual machines that run the stock NVIDIA driver, while the host keeps using it. It is a QEMU device that shows a guest an emulated NVIDIA GPU. The guest loads NVIDIA's unmodified driver on Windows or Linux; kayfabe reads what that driver asks for and has the real host GPU do the work, from an unprivileged host process. It is written in Rust.

This manual is for developers and system administrators who want to build, run and deploy it. It describes what exists in the repository today and says so plainly where something does not exist yet.

Status: read this first

• Research stage. It runs real applications, but there is no installer, no stable interface and no security audit.
• No public binary ships. You build it from source. A QEMU binary built from this tree is not yet distributable under the GPL (see Credits and notices).
• The security model is designed but not audited. What a guest-triggered GPU fault does to other VMs sharing the card is not yet measured. See Security model.
• Windows sleep/wake does not survive, Edge can reset after 7 to 16 minutes of playback, and other GPU dies need more work under Windows.
• Master is the current working version; newer experiments live on branches.

Nothing is patched: no host or guest driver or kernel is modified, and the guest uses NVIDIA's own installer. Kayfabe ships no NVIDIA or Microsoft binaries.

What has been run

These are the repository's own claims, with their limits. Each page of this manual repeats the evidence for what it describes.

• Windows 11 guest (Pro and Enterprise LTSC, NVIDIA's stock 580.88 driver): desktop, games, benchmarks, video decode, CUDA and professional applications on an RTX 4070 (AD104). A few registry settings are required: see Windows guest.
• Linux guest (NVIDIA's stock driver, nothing to set): a 30-arm test suite, a desktop with display planes, suspend/resume, and 61 of 65 CUDA applications with outputs equal to bare metal on a rented RTX 3060. The 4 failures use unified (UVM) memory paging and fail the checks; one of them printed SUCCESS with a wrong result, so check numerical output and host faults, not only the exit status. See Linux guest.
• GPUs: the families from Turing to Blackwell have been run in some form. The detail is on Requirements.

Pages

How to read the evidence labels

• [measured] means a run on named hardware at a named revision.
• [inferred] means reasoning, not a measurement.
• By design marks intended behaviour that no run or audit has confirmed. Not measured yet means no measurement exists; the project keeps one list of those on the site's What works today page.
• A sentence with none of these marks is a claim of fact and names its source or its evidence.
• not documented yet means the repository does not state a verified answer. The manual does not guess.
• Every page ends with a Verified on line. Where the repository's evidence does not cover a page end to end, it says Not verified end to end.

Where else to look

• Repository README: the landing text and current status.
• Design and internals: design documents, status records and evidence. They are for contributors, are not part of this manual and change faster than it does.
• To report a suspected vulnerability, see Security model.

Verified on: the status lines are copied from the repository README of 2026-10-21 (master 89fc8a222). Not verified end to end: this page is a map, not a procedure.

Requirements

This page lists only what the repository's evidence states. Where the repository gives no answer, the line says not documented yet. Nothing here is a promise of support: kayfabe is a research project.

Host

Operating system and CPU

• Linux on x86-64 with KVM (the host OS is Linux only). The device is a QEMU device (only QEMU is supported today; independence from one hypervisor is a goal, see Goals and plans), and every recorded run used /dev/kvm with -cpu host.
• The CPU needs hardware virtualisation (vmx or svm).
• Hosts that are themselves KVM guests (nested virtualisation) work, and most hardware results come from such rented hosts. A VM exit costs about 10 to 40 times as much on them as on bare metal. Results on non-nested hardware: the author's RTX 4070 host runs the Windows results in the README.
• Intel and AMD (EPYC 7452) hosts appear in the recorded results.
• Host kernel version: the recorded rented hosts ran Linux 6.8.0-59. A minimum or maximum supported kernel is not documented yet.
• arm64: no run on arm64 is documented. CI only cross-checks that the workspace builds for aarch64.
• An IOMMU is not required for kayfabe's isolation: the guest never drives the physical GPU. See FAQ.

NVIDIA driver on the host

• The open NVIDIA kernel module (-m=kernel-open) is what every recorded run used, and the provisioning script refuses a host whose /proc/driver/nvidia/version does not say "Open Kernel Module".
• Closed (proprietary) host modules: supported in principle, untested.
• Host driver versions that passed the GPU gates (9/9), the 30-arm thin-guest suite and the CUDA ladder with a 580.159.04 guest: 570.148.08 (kayfabe 1f3e4fc4), 575.57.08, 580.65.06 and 580.95.05 (ee35ca4a) and 580.159.04 (1837166d) [measured, GA102 boxes, September 2026; nested rented hosts].
• 595.91.07 (open) on the host and in the guest: gates 9/9 plus gate 10, bare metal 30/30 and thin guest 30/30 on one GA104 box [measured, kayfabe 714ec29d3, 2026-10-24, n=1]. The author's RTX 4070 host runs the same driver.
• Host drivers 565.57.01 and 550.54.14 were not swapped successfully in the recorded walk. The 545.x host attempt was blocked because its kernel module did not build on Linux 6.8 [code reading: docs/design/V3_DRIVER_MATRIX.md section 7, host-axis table]. No 535 host hardware result is recorded. Treat hosts older than 570 as untested.
• The host driver version is measured by kayfabe from the host, never taken from a guest. An unmeasured host driver is refused by name.
• libcuda (the CUDA driver library) must be loadable on the host: kayfabe opens it with dlopen to run its page-table walker on the GPU. The walker's PTX is committed in the repository, so no CUDA toolkit is needed to build.

GPU

Kayfabe reads the host GPU's facts from the host driver. Marketing names do not decide the die: the RM architecture and implementation numbers do.

What has been run (the README's list; revisions and boxes are in the repository's status records):

• Earlier revisions also passed Linux on GA102, GB203 and a floor-swept RTX 3060 Ti.
• Hopper, GA100 and datacentre Blackwell (GB10x) are derived from the open driver's source only; no hardware was available. GA100 and GB10B are refused by name.
• Pre-Turing GPUs cannot work. Below Turing there is no GSP firmware interface for kayfabe to impersonate: the guest driver would drive the hardware directly.
• Consumer GPUs are the tested class; "any NVIDIA GPU" is the goal, not a claim.
• The table above is a map of historical and branch-only results, with those scopes stated. A larger inventory of recorded family evidence is in the internal design document docs/design/V3_FAMILY_COVERAGE.md (merged 2026-10-28). Neither table establishes that every die passes on current master.

Memory and apertures

• Video memory: the guest's video memory is a real allocation of host VRAM, sized by the device property fb-mb (default 8192 MiB). On a card with less than 10 GiB the bench harness sets it to (total VRAM minus 2048 MiB) rounded down to whole GiB, and the harness script records that 6144 MiB boots on an 8 GiB card. A store the card cannot hold is refused at start with store of N MiB refused: NoMemory.

• Host BAR1: the sum, over the kayfabe devices on one card, of guest BAR1 + guest BAR2 + 1 MiB (PRAMIN window) + 16 MiB of headroom must not exceed the host's BAR1 aperture. With the common 256 MiB host aperture, a guest BAR1 of 256 MiB does not fit and the harness uses 128 MiB. The device refuses at start, by name, a configuration that does not fit. A larger, resizable host BAR1 is not required for that reduced guest aperture; the required host aperture follows from the stated budget [code reading: crates/kf-qemu/src/cardbudget.rs, scripts/fastguest/run_fast_guest.sh].

• Guest RAM must be a shared memfd (memory-backend-memfd,share=on); with ordinary anonymous RAM every store-backed path refuses.

• Host RAM and disk: the bench's own host checks demand at least 16 GB RAM, 8 cores and 100 GB free disk, but those are limits for a build-and-test box, not product requirements. The harness defaults for a Windows guest are 8192 MB and 8 vCPUs, with an 80 GB disk. A product minimum is not documented yet.

• CPU-window admission: each CPU window permits at most 8192 live guest placements across RAM and store, plus its separate boot seed. Further placements are refused by name; successful unmap returns capacity.

Privileges

• The design runs the host side from an unprivileged process using only resource-manager controls the host driver marks non-privileged.
• The provided launch scripts, however, run as root: they open /dev/nvidia*, create a tap network and take a lock.
• One run without root is recorded, on one host: a user whose only group was kvm (no capabilities) ran the 30-arm Linux suite 30 of 30, the GPU gates 9 of 9 and a Windows 11 guest running Unigine Heaven, with no VFIO rebind and no kernel module, driver or udev change [measured, kayfabe 0a746117, RTX 4070, driver 595.91.07]. It relied on NVIDIA's default mode 0666 for /dev/nvidia*, and the user needed permission to open /dev/kvm. The record is on the branch claude/rootless-audit-20261016 (traces/rootless_audit_20261016/REPORT.txt), not on master; a repeatable test on master is still to be added. See Security model.
• No host patches are required. An optional patch to the host's nvidia-uvm exists as research; it is not part of the stock path. Four of the 65 CUDA applications need unified-memory demand paging and fail without it. Optional enhancements (a guest helper module, patched guest or host drivers) are planned and never required.

Build host

See Building from source for the package list, the pinned Rust toolchain and QEMU 10.2.4.

Guest

Linux

• NVIDIA's stock driver, unmodified, from the .run installer or the distribution package. Nothing to set.
• Every recorded Linux run used the open kernel module flavour in the guest. Other flavours are not documented yet.
• Guest driver versions that passed the CUDA ladder with host 580.159.04: 575.57.08, 580.105.08, 580.159.04, 590.48.01, 595.84 and 610.57.04 [measured, September 2026, GA102 box]. A declared guest version outside the measured set is refused by name.
• The 30-arm thin-guest suite is graded for 580.x guests only.
• Display with the guest's own display driver: the wire layouts were derived for guest 580.159.04 only (as of 2026-09-30); any other guest version is refused. Check the current state before relying on another version.

Windows

• Windows 11 Pro (build 26200) and Windows 11 Enterprise LTSC 2024 (build 26100) with NVIDIA's stock 580.88 driver, plus three registry settings. See Windows guest.
• The Windows harness normally gives the guest a TPM 2.0 (swtpm), Secure Boot firmware (OVMF) and kayfabe's boot display driver. A headless device without display or boot-display driver reached Code 0 before resets and a later loss of guest responsiveness [measured at revision b439ae82, lease 57908255, GA106, guest 580.88; branch claude/tur1-20261027, not merged, traces/tur1_20261027/iter2_ga106/ga106_h1_ndtimeline.csv]. That establishes initialisation without the boot-display driver, not a stability pass. A kayfabe guest without TPM or Secure Boot has no recorded acceptance run.
• You supply and license Windows yourself.

Not supported or not finished

• Windows sleep/wake.
• GA100, GB10B (refused by name); Hopper and GB10x (never run on hardware).
• Multi-GPU: the two-GPU thin-guest record covers copy-engine checks, serially and concurrently; the full 30-arm regression was a separate, single-device run [measured in docs/design/V3_MULTI_GPU_AUDIT.md section 7, revision 74fced87, lease 52664396, two RTX 3060s on one host]. CUDA inside a two-GPU guest, UVM across two GPUs and peer access were not measured.
• Several VMs sharing one GPU as a verified, isolated configuration: the effect of a guest-triggered GPU fault on other VMs is not yet measured.

Verified on: the lists above are taken from the repository README (2026-10-21), the GPU sweep record of 2026-10-24 (kayfabe 714ec29d3, one GA104 box) and the driver-matrix record of September 2026 (kayfabe ee35ca4a, 1f3e4fc4, 1837166d; GA102 boxes). Not verified end to end: no single machine was checked against this page as a whole.

Building from source

There is no installer and no public binary. You build two things: the Rust workspace, and a QEMU binary that has the kf3-gpu device linked in. This page lists the steps the repository's build scripts perform, in their order.

Read Requirements first. A QEMU binary built from this tree is not yet distributable under the GPL: see Credits and notices.

1. Get the source

git clone https://github.com/reindertpelsma/kayfabe.git
cd kayfabe

The submodules under third_party/ are reference sources. They are not built, linked or copied, and a normal clone does not initialise them. You do not need them to build.

2. Install build packages (Debian/Ubuntu)

The package policy lives in one function of scripts/bench/provision_box.sh, and you can print the list instead of reading the script:

bash scripts/bench/provision_box.sh --packages full    # Rust workspace build
bash scripts/bench/provision_box.sh --packages bench   # QEMU build and the guest disk tooling

At the time of writing, full is build-essential pkg-config git curl python3 kmod pciutils cpio zstd psmisc busybox-static openssh-client clang lld libssl-dev and bench is ninja-build meson python3-venv python3-pip python3-tomli libglib2.0-dev libpixman-1-dev flex bison qemu-utils cloud-image-utils genisoimage qemu-system-x86 ovmf musl-tools libslirp-dev. The script is authoritative.

3. Install the pinned Rust toolchain

The repository pins one toolchain in rust-toolchain.toml (channel 1.99.0, with rustfmt and clippy) and lists two extra targets: x86_64-unknown-linux-musl and x86_64-unknown-uefi.

curl -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain stable --profile minimal
. "$HOME/.cargo/env"
rustup toolchain install 1.99.0 --profile minimal
rustup target add --toolchain 1.99.0 x86_64-unknown-linux-musl x86_64-unknown-uefi

(rustup normally adds the listed targets itself when it installs the pinned toolchain; the commands above make it explicit, as the provisioning script does.) Without the musl standard library the whole workspace fails with error[E0463]: can't find crate for std on a crate that has nothing to do with musl. The UEFI target is needed because the build compiles kayfabe's boot-display driver (firmware/kf-gop) from source; a missing target is a named build error. No compiled binary is committed to the repository.

4. Build and test the workspace

cargo build --workspace --release
cargo test -p kf-oprom          # any kf-* crate; no GPU needed

• cargo test -p kf-<crate> runs without a GPU for the kf-* crates.
• crates/kayfabe-* is the frozen earlier tree. It is built by --workspace and kept building because the test harness's raw client depends on it.
• The shipped crates have one external crate in their dependency closure (libc).

5. Build QEMU with the kf3-gpu device

The device is a small C QOM device (qemu/hw/misc/kf3/) plus a Rust static library (crates/kf-qemu). build_kf3.sh lays the first into a QEMU source tree, links the second, and builds QEMU in its own build directory.

The scripts fetch QEMU 10.2.4. Other versions are not documented.

curl -fsSL -o qemu-10.2.4.tar.xz https://download.qemu.org/qemu-10.2.4.tar.xz
tar -xf qemu-10.2.4.tar.xz
bash scripts/bench/build_kf3.sh "$PWD/qemu-10.2.4" [<build-dir>]

cargo must be on PATH. The build directory defaults to qemu-build-kf3 beside the QEMU source directory. What the script does, in order:

1. Runs cargo build --release -p kf-qemu and requires the resulting target/release/libkf_qemu.a. A failed cargo step stops the script, so a stale archive is never used.
2. Copies every qemu/hw/misc/kf3/*.c, *.h and meson.build into <qemu>/hw/misc/kf3/, and the archive beside them.
3. Appends subdir('kf3') to hw/misc/meson.build and a config KF3 stanza (bool, default y if TEST_DEVICES, depends on PCI) to hw/misc/Kconfig, unless they are already there.
4. Takes a lock on the build directory, then configures QEMU, only if the directory is new or was configured with different flags (a stale configuration is removed whole and rebuilt):

   --target-list=x86_64-softmmu --disable-docs --disable-tools --disable-guest-agent
   --disable-werror --enable-slirp --enable-vnc --enable-pixman --disable-gtk --disable-sdl
   --disable-curses --disable-libssh --disable-vde --enable-tpm --without-default-features
   --enable-kvm --enable-system
   

   --enable-pixman and --enable-vnc are there because the device registers a QEMU graphic console per virtual head. --enable-tpm is for Windows 11 guests (a software TPM 2.0); --enable-slirp is for the Windows harness's user networking.
5. Runs ninja -C <build-dir> qemu-system-x86_64.
6. Installs the binary privately per revision at <build-dir's parent>/kf3-bins/<git rev>[-dirty]/qemu-system-x86_64 (atomic rename), with pc-bios and qemu-bundle linked beside it, and prints a KF3_BUILT <path> rev=<rev> ... line.

The per-revision directory exists because a shared binary was once replaced under a running test. The launch scripts run the binary for their own checkout's revision.

6. Optional builds

The GPU gates (kf-gate1 to kf-gate9) play the guest against the real planes on a real GPU with no QEMU: cargo build --release -p kf-harness --bins, run through scripts/bench/v3_gates.sh (see Running the QEMU device).

The raw client is the test program the 30-arm suite uses, built twice: for the host (bare-metal baseline) and as a static musl binary for the guest.

cargo build --release -p kayfabe-rm-ladder --bin kayfabe-rm-ladder
cargo build --release --target x86_64-unknown-linux-musl --bin kayfabe-rm-ladder

The thin guest (a kernel and an initrd that insmods the driver, runs the raw client and powers off): bash scripts/fastguest/build_fast_guest.sh [guest.qcow2] [outdir]. It needs busybox and cpio, a guest image that is not in use, and the raw client.

The display broker (the host program that owns the window and input; see Display and the broker):

cd broker && make        # prints which backends it kept
make check               # build + selftest

Build dependencies named in broker/Makefile: libwayland-dev wayland-protocols (Wayland backend), libxcb1-dev libxcb-dri3-dev libxcb-present-dev (X11 backend), optionally libxcb-render0-dev and libxcb-xinput-dev. A backend is built only when its libraries are present, and the build prints which ones it kept.

Not documented yet

• A one-command install, distribution packages, containers or a prebuilt binary.
• Building against a QEMU other than 10.2.4, or against a distribution's QEMU packages.
• Cross-building for arm64.
• Windows-host builds.

Verified on: cargo test -p kf-oprom was run for this page (Rust 1.99.0, no GPU; every test binary of that crate passed). build_kf3.sh, cargo build --workspace --release, the broker build and the thin-guest build were read from their scripts, not run for this page. Every hardware result in the repository was produced with binaries from build_kf3.sh.

Running the QEMU device

There is no installer. Kayfabe is started today by the repository's own bench scripts, which were written for rented, disposable test hosts and run as root. This page describes what those scripts do and the QEMU device they start, so you can reproduce it on a machine you dedicate to the purpose. Every command names the file it comes from.

Warning. The provisioning scripts purge NVIDIA driver packages, stop the host's display manager, load and unload kernel modules, create a tap network device, change ip_forward and add iptables rules. They are written for throwaway test machines. Do not run them on a machine you care about; read them first.

Overview

1. Put the open NVIDIA driver on the host.
2. Build the QEMU binary with the kf3-gpu device (Building from source).
3. Prepare a guest (Linux guest, Windows guest).
4. Start QEMU with a kf3-gpu device. One GPU job at a time.

1. The host driver

The recorded runs use the open kernel module. The script that installs it for the bench is scripts/bench/provision_host_driver.sh (run as root; default version 580.159.04):

HOST_DRIVER=580.159.04 bash scripts/bench/provision_host_driver.sh

What it does, in order: blacklists nouveau; un-holds and purges every nvidia-* / libnvidia-* package; stops a running display manager (display-manager, sddm, gdm3 or lightdm) and nvidia-persistenced; unloads the NVIDIA modules; removes the dkms tree; installs the compiler the running kernel was built with if needed; runs

sh NVIDIA-Linux-x86_64-<version>.run --silent --no-x-check --no-nouveau-check --no-questions --dkms -m=kernel-open -j8

then loads the module and, if nvidia-smi does not name the GPU, tries up to three unload / PCI reset / reload recoveries. The .run comes from https://us.download.nvidia.com/XFree86/Linux-x86_64/<version>/, with the tesla/<version>/ path as a fallback.

Check the result on content, not on the installer's exit status: /proc/driver/nvidia/version must say "Open Kernel Module".

2. The bench tree

scripts/bench/provision_bench_tree.sh (run as root after the driver step) prepares everything under /workspace/bench (override with BENCH_DIR in the launchers):

• installs the packages of provision_box.sh --packages bench;
• downloads QEMU 10.2.4 and the Ubuntu 24.04 (noble) cloud image;
• creates a tap device nvktap0 with host address 192.168.77.1/24, enables IPv4 forwarding and adds a masquerade rule and two FORWARD accepts for it; the guest sits at 192.168.77.2;
• generates an ssh key guest_key;
• builds the kayfabe QEMU with build_kf3.sh (into qemu-build-kf3, and kf3-bins/<rev>/);
• builds a guest disk guest.qcow2 (the cloud image plus 30 GB), seeds it with cloud-init, boots it on stock QEMU, and installs the guest driver (see Linux guest);
• builds the guest-side raw client (musl).

The script expects the guest driver .run of the same version as the host at /root/NVIDIA-Linux-x86_64-580.159.04.run; that version string is fixed in the script.

The tap network is the bench's choice. The Windows harness uses user-mode networking instead, so it needs no root network device.

3. One GPU job at a time

The launch scripts take an exclusive lock, /tmp/kayfabe-fastguest.lock (override with KF_LOCK), and wait for it. GPU runs are strictly serial: two concurrent runs of the fast lane kill each other's QEMU.

Between runs the launcher also waits (up to about 30 s) for the previous VM's GPU memory to drain (KF_GPU_IDLE_MIB, default 512).

4. The kf3-gpu device

Add the device to a QEMU command line built from the binary in kf3-bins/<rev>/. The device is a PCI display device. It takes its identity (vendor and device ids, the die's facts) from the host GPU at start.

Machine arguments it needs

-object memory-backend-memfd,id=ram0,size=<N>M,share=on
-machine q35,accel=kvm,memory-backend=ram0 -m <N> -cpu host

Guest RAM must be a shared memfd. With plain -m every store-backed path refuses. Use -vga none when the device provides the display.

Device properties

Every property below is declared in qemu/hw/misc/kf3/kf3.c. Defaults are as declared there.

Properties are given as -device kf3-gpu,fb-mb=8192,bar1-size=134217728,bar2-size=33554432,id=kf0, or for one property on an existing device -global kf3-gpu.<property>=<value>.

Environment variables beginning with KF3_ also exist (for example KF3_RPC_TRACE=1 logs every command the emulated GPU answers). They are for development and benchmarking. A complete, supported list is not documented yet; the harness sets what it needs.

Example: the Linux desktop guest's device line

Assembled from scripts/bench/display/interactive.sh (not run as written for this page):

-device kf3-gpu,id=kf0,guest-driver=580.159.04,fb-mb=8192,bar1-size=134217728,bar2-size=33554432,display=on,gop=on,display-broker=<socket>,display-broker-uid=<uid>

(The script also sets x11-dispsw=on; see Display and the broker.)

5. Checks you can run

All of these assume the bench tree above. They are strictly serial, one box at a time.

Rules the harness applies, which are worth copying to any setup of your own:

• A timeout is a crash. The thin guest gets one hard budget for the whole run (QEMU start to power-off).
• A green run must have reached hardware. The thin guest run fails if a guest channel was never forwarded to the GPU (KF_REQUIRE_FORWARD=0 turns that off for a deliberate control run).
• If bare metal passes and the guest fails, the defect is kayfabe's. Run the host baseline first.
• Watch the host's kernel log for Xid messages across a run. A host GPU fault does not show up in the guest. bash scripts/bench/host_xid_watch.sh <tag> -- <command> runs a command and exits 6 if a new host Xid appeared across it.
• Write a start marker and an exit line for every job; a killed job and a running one look alike otherwise.

6. Stopping a guest

The launchers stop a guest by asking it to power down (ACPI), then waiting, and only then killing QEMU. The interactive Linux launcher uses scripts/bench/display/interactive.sh stop; the Windows harness uses scripts/bench/windows/win_vm.sh stop (guest shutdown, then QMP system_powerdown, then QMP quit, never SIGKILL).

Not documented yet

• A supported rootless installation guide. The branch claude/rootless-audit-20261016 has launch notes and permission checks in sections 1, 2 and 7, with scripts under scripts/bench/rootless/. That audit is not merged, and its host-specific launchers are not a general installer. The scripts described here still run as root. See Security model.
• libvirt, systemd units, container images and sandbox profiles. The design leaves sandboxing the (single) QEMU process to the deployer: see Security model.
• A complete reference for the KF3_* environment variables.
• Several VMs on one GPU as a supported configuration.

Verified on: the scripts and properties above were read from the repository at master (04773c348). Their results are recorded in the repository for kayfabe afb552ea (gates 9/9, bare metal 30/30, thin 30/30, RTX 3060, 2026-09-30) and 714ec29d3 (GA104, 2026-10-24). Not verified end to end for this page: no script on it was run.

Linux guest

A Linux guest uses NVIDIA's stock driver, unmodified, and nothing has to be set in the guest: no registry-style settings, no guest patches, no custom kernel module.

What has been run

All of these are the repository's claims, with the revision or box where it states one.

• The 30-arm suite passes in the thin guest (also with the relay forced on), with the GPU gates at 9/9. [measured: kayfabe afb552ea, RTX 3060, 2026-09-30; GA104 at 714ec29d3, 2026-10-24]
• CUDA is correct: cup3 returns 43, cup8 (a 2048 x 2048 matrix multiply) returns BAD=0 MAXERR=0.
• 61 of 65 CUDA applications pass (CUDA samples, PyTorch, Hugging Face generation, llama.cpp), with 6 of 6 stream probes and 100 processes in one boot, and outputs equal to bare metal [measured, an earlier revision, a rented RTX 3060]. Four workloads need unified (UVM) memory paging and fail correctness checks, including CUDA error 719, a host Xid 31 and a recorded wrong result despite SUCCESS.
• NVENC and NVDEC output is byte-identical to bare metal in the graded lane [measured, RTX 3060].
• Headless graphics: Vulkan, EGL, and GLX through Xvfb and VirtualGL; 38 of 38 items of the test set, 31 byte-identical to bare metal [measured, GA104].
• A desktop with display planes (overlay, YUV) and 3 of 3 suspend/resume cycles. See Display and the broker.
• Driver versions: see Requirements.

Not done: demand-paged managed memory (4 applications), a rootless boot record on master (one exists on the branch claude/rootless-audit-20261016, see Security model), and any statement about sharing performance between two VMs on one GPU (two VMs have booted together; see the FAQ).

Prepare a guest image

The bench builds its guest image once, from an Ubuntu 24.04 (noble) cloud image, on stock QEMU (no kayfabe), then reuses it. These are the steps scripts/bench/provision_bench_tree.sh performs. The bench scripts assume the host-mode driver: the same .run as the host.

qemu-img convert -O qcow2 noble-server-cloudimg-amd64.img guest.qcow2
qemu-img resize guest.qcow2 +30G
cloud-localds seed.iso user-data meta-data      # an ubuntu user, your ssh key, a static address

Boot it once on stock QEMU (-enable-kvm -cpu host -m 8G -smp 8, two virtio drives, user networking with an ssh forward), wait for cloud-init to finish, then in the guest:

sudo apt-get install -y build-essential linux-headers-$(uname -r)
# copy NVIDIA-Linux-x86_64-<version>.run to the guest, then:
sudo sh nv.run --silent --no-x-check --no-nouveau-check --no-questions -m=kernel-open -j8
modinfo nvidia | grep -E '^version|^license|^vermagic'     # check the version on content
sudo poweroff

Notes from the script:

• The guest needs a C compiler and kernel headers for the running kernel; if the installer's log stops at "Kernel module compilation complete" with no error, the driving ssh session was probably killed mid-install and the modules were built but never installed.
• All recorded runs use -m=kernel-open, the open kernel module.
• The script also installs a cuda.h into the guest for the CUDA test programs (the CUDA runtime archive, version 12.6.77).
• The cloud-init seed in the script gives the guest the address 192.168.77.2/24 on the host's nvktap0 network. Edit it if your network differs.

For other guest driver versions the driver-matrix scripts build one guest image per version (scripts/drivermatrix/stage_fat_guest.sh); the launcher takes the image from KF_GUEST_IMG.

Boot it on kayfabe

Use a QEMU built by build_kf3.sh, and start it with the device as shown in Running the QEMU device. The bench's launcher for a full guest is scripts/bench/boot_nvkvm.sh <tag> [extra qemu args...], run in the foreground; scripts/bench/boot_capture.sh <tag> boots and also stores the guest driver's own dmesg and the host driver's log for the boot in files.

What the launcher sets, from its source:

• shared memfd RAM and -machine q35,accel=kvm (2048 MB unless NVKVM_RAM_MB), -cpu host, -smp 3 unless KF_SMP;
• a virtio disk from KF_GUEST_IMG (default /workspace/bench/guest.qcow2) and a virtio-net on the nvktap0 tap;
• BAR1 of 128 MiB (KAYFABE_GUEST_BAR1_MB), BAR2 of 32 MiB, and fb-mb from KF3_FB_MB, else 8192 MiB, or (VRAM - 2048 MiB) on cards under 10 GiB;
• KF_FIRMWARE=ovmf for UEFI firmware instead of SeaBIOS (needed by the boot display, whose option ROM is a UEFI driver).

Then, inside the guest, the driver loads as it would on hardware. nvidia-smi shows the single GPU, which the host keeps using.

The guest driver's dmesg is not in the serial log unless the harness captures it, and a serial console is slow: verbose tracing over the 16550 UART once consumed most of a run's time budget, so the harness uses a virtio console.

The thin guest (a test environment, not a desktop)

For testing, a guest with no disk and no services exists: a kernel and initrd that load the driver, run the raw client and power off. Results come over the serial console. Build it with scripts/fastguest/build_fast_guest.sh and run it with scripts/fastguest/run_fast_guest.sh. See Running the QEMU device for the checks.

More than one GPU

One kayfabe device is created per host GPU. KF3_GPUS=0,1 makes the thin-guest harness add one kf3-gpu,gpu-minor=<m> per listed host minor. A thin guest with two distinct host GPUs passed copy-engine checks serially and concurrently [measured in docs/design/V3_MULTI_GPU_AUDIT.md section 7, revision 74fced87, lease 52664396, two RTX 3060s on one host]. The full 30-arm regression was a separate single-device run. CUDA inside a two-GPU guest, UVM across two GPUs and peer access were not measured.

Not documented yet

• A supported way to build a guest image on other distributions.
• A supported guest-kernel range. Recorded examples include the thin guest on 6.8.0-59 and the full guest on 6.8.0-139. The branch codex/sweep-integrate-20261018, not merged, also records 6.8.0-147-generic in a GB202 desktop attempt whose presentation failed [measured at revision 24906bf33, lease 56788212; traces/sweep_integrate_20261018/phase2_blackwell_56788212/B2_RESULT.md on that branch]. That extends the record; it is not a support promise.
• The proprietary (closed) guest module.
• A guest-side guide for container runtimes.

Verified on: Linux guest results are recorded for kayfabe afb552ea (RTX 3060 GA106, 2026-09-30) and 714ec29d3 (GA104, host and guest 595.91.07, 2026-10-24); the application matrix is from an earlier revision on a rented RTX 3060. Not verified end to end: the image-preparation steps above were read from the script, not run for this page.

Windows guest

What a Windows 11 guest needs to run NVIDIA's stock driver on kayfabe. Labels: [measured] = a run on named hardware, [inferred] = reasoning.

What was tested

• Windows 11 Pro, build 26200 (the full retail edition) [measured, 2026-10-14, kayfabe eb9f2aa3, RTX 4070]: the desktop and the application matrix (the benchmark output names "Microsoft Windows 11 Pro", kernel 10.0.26200).
• Windows 11 Enterprise LTSC 2024, build 26100 (Microsoft's 90-day evaluation) [measured]: the image that scripts/bench/windows/win_vm.sh install builds, and the one used on rented hosts.
• Driver: NVIDIA's stock Windows driver, version 580.88, installed with NVIDIA's own installer or INF, unmodified. No driver, kernel or Windows binary is patched.
• Applications run unmodified from the vendors' complete installers; the README's table lists the results (Minecraft, Unigine Heaven, GravityMark, Edge with hardware video decode, CUDA, Blender, OBS, Office, SolidWorks and 3DMark). Several of those rows say their result files are not committed yet.

The settings it needs (and why)

The stock GeForce driver on Windows runs its resource manager on the CPU by default. Kayfabe serves only the GSP firmware path, so the driver has to be told to use GSP. Without the first setting the driver stops with Code 43 [measured on Windows 11 and on LTSC]. All values are DWORDs and take effect after a restart.

From an elevated command prompt in the guest. First read the display adapter's driver key, then substitute it:

powershell -NoProfile -Command "(Get-PnpDeviceProperty -InstanceId (Get-PnpDevice -PresentOnly | Where-Object InstanceId -like 'PCI\VEN_10DE*').InstanceId -KeyName DEVPKEY_Device_Driver).Data"
reg add "HKLM\SYSTEM\CurrentControlSet\Control\Class\<the key printed above>" /v EnableGpuFirmware /t REG_DWORD /d 1 /f
reg add "HKLM\SYSTEM\CurrentControlSet\Services\nvlddmkm" /v EnableGpuFirmware /t REG_DWORD /d 1 /f
reg add "HKLM\SOFTWARE\Microsoft\Windows\Dwm" /v OverlayTestMode /t REG_DWORD /d 5 /f
shutdown /r /t 0

scripts/bench/windows/gsp_on.ps1 does settings 1 and 2 while it installs the driver (it is what win_vm.sh nv-install runs): it disables the device, adds and installs the INF, writes both values, reads them back, then re-enables the device. Setting them before the driver's first start avoids the Code 43.

These are settings for a development guest, not a complete installation guide for a production system.

The Windows harness: win_vm.sh

scripts/bench/windows/win_vm.sh installs a Windows 11 guest once and then boots it on kayfabe as many times as you like. It is one script and it needs a bench tree and a kayfabe QEMU built for the checkout's revision (Building from source, Running the QEMU device). It refuses by name when a command or file it needs is missing.

Install the host tools it names:

apt-get install -y ovmf swtpm swtpm-tools sbsigntool python3-virt-firmware xorriso netpbm p7zip-full

What it sets up

Commands

S=scripts/bench/windows/win_vm.sh
$S install                                  # once; about an hour on a nested box
$S run --detach --tag basic                 # kayfabe + signed boot-display driver, display on
$S ssh 'Get-PnpDevice -Class Display'
$S shot                                     # PNG of the console
$S check --kf3                              # WIN_* lines: Secure Boot, db, TPM, BitLocker, virtio...
$S nv-install                               # NVIDIA 580.88, GSP forced on before nvlddmkm's first start
$S collect nv1                              # Windows-side evidence into logs/evidence/nv1/
$S stop                                     # guest shutdown, then QMP powerdown, then QMP quit
$S status

Other sub-commands in the script's header: scp-to, scp-from, sign (export and sign this checkout's boot-display driver) and fetch (download and verify the pinned media).

run options:

The harness environment: WINVM_ROOT (default /workspace/winvm), WINVM_NAME, WINVM_RAM_MB (8192), WINVM_SMP (8), WINVM_KF3_FB_MB (8192), WINVM_SSH_PORT (2224), WINVM_QEMU.

The QEMU command it builds uses the pc-q35-10.2 machine with SMM, Secure Boot firmware, the TPM, and the device kf3-gpu,fb-mb=8192,bar1-size=134217728,bar2-size=33554432,display=on,gop=on,gop-efi=<signed driver>,id=kf0. Every guest reboot starts a fresh QEMU with the same firmware variables and the same TPM state; the harness restarts it for you.

Rules the script enforces: every step logs START and EXIT rc= (a log with no WINVM_RUN_END belongs to a dead run); GPU work takes the shared lock /tmp/kayfabe-fastguest.lock; nothing secret is written into the repository. Keep the swtpm state and the ssh key private; a VM is never restored from another VM's state.

Remote access: ssh -J <gpu-host> -p 2224 kf@127.0.0.1 with a key you passed to install --authorized-key. To watch the screen, forward the VNC socket: ssh -N -L 5905:/workspace/winvm/kfwin/run/vnc.sock <gpu-host> and point a VNC viewer at localhost:5905.

Other things a Windows guest needs (not settings)

• A unique SMBIOS UUID, serial and MAC per VM, and offline setup until after OOBE (Windows Autopilot once matched a default-identity VM to an unrelated company tenant); an explicit gpu-uuid.
• Do not snapshot an image with a Windows update staged: it crash-loops dwm.exe. Block updates in test images.
• You supply and license Windows. The scripts download Microsoft's evaluation image, the virtio drivers and NVIDIA's installer from their vendors, each pinned by SHA-256. See Credits and notices.

Known limits

• Windows sleep/wake is not supported yet.
• Edge can reset the display driver after 7 to 16 minutes of playback.
• Other GPU dies need more work under Windows; GB206 under Windows is open (Code 43).
• nv-install is slow: it copies the 2.6 GB driver package over user networking at about 2.4 MB/s (18 minutes).
• A failed NVIDIA driver slows everything down. With the driver in Code 43, boot to ssh took about 20 minutes and the guest ignored shutdown /s for 5 minutes.
• The guest changes its own Secure Boot variables (it adds the 2023 CAs to db and updates dbx). The variables file is per-VM live state; never regenerate it or copy it to another VM.

See also Troubleshooting and Display and the broker.

Not documented yet

• A Windows install on a machine without the harness (a plain ISO install onto a kayfabe-backed VM by hand).
• Windows 10, or other Windows 11 builds than the two above.
• A successful guest run with another NVIDIA driver version. Windows 581.51 was attempted on AD102 and failed with Code 43 [measured in the sweep matrix, revision 24906bf33, P19 profile; historical source 9d6f210ba]. This is recorded on the branch codex/sweep-integrate-20261018, not merged: matrix. Its same-board native Windows control passed, which is not a kayfabe guest pass.

Verified on: Windows 11 Pro 26200 with driver 580.88 on an RTX 4070 (kayfabe eb9f2aa3, 2026-10-14; further results up to master 89fc8a222, 2026-10-21, in the README); LTSC 26100 on rented hosts. Not verified end to end: the reg add snippet is the repository's own text and is not a recorded run of exactly those lines. Settings 1 and 2 are the part that gsp_on.ps1 applies and reads back in the harness.

Display and the broker

A guest can run headless (compute and offscreen rendering need no display), or kayfabe can give it a virtual monitor. This page covers the second case and the host program that puts the picture on your desktop.

What the display feature is

• Opt-in. The device property display=on makes the guest's own NVIDIA display driver drive a virtual NVIDIA display. The default is off.
• The guest's driver programs the display; kayfabe decodes those display channels itself and scans the result out. Display channels have no GPU work behind them: they are emulated channels.
• What has been run: a pixel-exact 1920 x 1080 scanout at 60 Hz; a Linux Mint Cinnamon desktop on Wayland with vkcube in a window; weston; Xorg with NVIDIA's X driver [measured 2026-09-30, Linux guest 580.159.04]. Linux display planes (overlay, YUV) and Windows 11 desktops [measured, see the README].
• The boot display (gop=on): a small UEFI driver that kayfabe carries in the card's expansion ROM, so the firmware and boot screens, the boot menu and the early console appear on the virtual monitor. It is compiled from source during the Rust build. For Secure Boot guests it has to be signed, and its path is given by gop-efi=.
• Per-guest-driver layouts: the display wire layouts were derived for Linux guest driver 580.159.04 only (as of 2026-09-30), and any other guest version is refused. Check before relying on another version.
• x11-dispsw (a display class that X11 desktops need) is experimental and off by default.
• Colour: only an opt-in, bounded SDR subset has been qualified; HDR and arbitrary colour pipelines are not claimed.

Ways to see the screen

The build in Building from source disables the GTK and SDL front ends. What is left:

1. VNC. QEMU registers a graphic console for each virtual head. The harnesses use -vnc: the Windows harness exposes it on a unix socket (-vnc unix:<run>/vnc.sock), and KF_VNC=1 in the interactive launcher adds -vnc 127.0.0.1:7 for view-only use. win_vm.sh shot takes a PNG with screendump.
2. The display broker, a separate program that opens a window on your desktop and owns its input. This is the path the repository's interactive launchers use.

The broker

The broker is a small program that owns the window, the input and the clipboard, so the VMM does not have to. QEMU hands it finished frames over a unix socket and holds no connection to your display server at all. It was imported from the author's nvkvm-pv repository and is built from broker/.

Why a broker, and not QEMU's own window? Kayfabe's QEMU is built without the GTK and SDL front ends on purpose (see above), so the broker is the way to get a desktop window; VNC stays as the second path. The reasons the repository records:

• No lock-in to one hypervisor. The owner's direction of 2026-10-01 is that frames leave through a hypervisor-agnostic broker, not QEMU's SDL or GTK UI, and hypervisor independence is a goal (Goals and plans). In the broker relay only about 230 lines in the QEMU device are QEMU-specific; the connection logic, the frame ring and the input policy are VMM-neutral Rust.
• The VMM stays away from your display server. QEMU holds no connection to it at all, which matters most when QEMU is confined (a container, systemd-nspawn, bubblewrap), the extra layer the Security model recommends.
• One display path to maintain. Pointer grab, absolute or relative input, buttons, the wheel and re-sync are decided in one place (kf-broker); only a thin shim names the VMM, so a second hypervisor adds a shim rather than a second display path.
• The broker owns the cursor and the pacing. It draws the guest's cursor, including true inverting cursors, and kayfabe composes none; the design makes the broker the frame clock (not built yet), with kayfabe's clock as the fallback after 50 ms of silence. The display design notes record problems with QEMU's own front ends: SDL and GTK read a guest cursor with different conventions, GTK warps the host pointer under a grab, and GTK's window resizing re-modes the guest.
• Frames are copied on the GPU into a frame kayfabe owns. For a compositor on the same NVIDIA GPU the frame never crosses to the CPU, and the guest cannot change it after the copy; other compositors get a copy in host RAM, with shared memory as the last fallback.

The imported broker/README.md still says to use -display sdl for an unconfined QEMU. That advice is nvkvm-pv's and does not apply here: this QEMU has no SDL.

Note that broker/README.md shows QEMU options of the nvkvm-pv project (-display nvkvm-broker,socket=...). Kayfabe does not use those. In kayfabe the socket is a property of the device: display-broker=<absolute socket path>.

Build and run

cd broker && make          # prints which backends it kept (Wayland, X11)
make check                 # build + selftest
nvkvm-display-broker --socket /run/user/<uid>/nvkvm/display.sock --backend auto --persist --title "kayfabe guest"

Run it in your desktop session as your normal user. Then give QEMU the same socket and your uid:

-device kf3-gpu,...,display=on,gop=on,display-broker=/run/user/<uid>/nvkvm/display.sock,display-broker-uid=<uid>

display-broker-uid names one more uid accepted as the broker; uid 0 and QEMU's own effective uid are always accepted, and the owner of the socket's directory is not.

Test the broker alone before involving QEMU: nvkvm-broker-testclient /run/nvkvm/display.sock --present 640x480 prints the handshake and every input event and sends one buffer.

Broker options (from the program's own usage text): --socket PATH, --socket-mode OCTAL (default 0600), --socket-group NAME, --backend auto|wayland|x11|test, --size WxH (default 1920x1080), --title TEXT, --allow-user NAME, --allow-group NAME, --drop-user WHO, --fullscreen, --scale aspect|stretch|none, --persist (keep the window when the VMM disconnects), --clipboard off|guest-to-host|consent, --clipboard-trigger LIST, --refresh auto|none|<Hz>, --resolution auto|none|WxH, --present-mode, --verbose. Run it with --help for the full list.

By default root and the invoking user may connect. The socket is mode 0600.

Grab, focus and hotkeys

• CTRL+ALT+G toggles grab: all keyboard is captured and the pointer is locked. CTRL+ALT+F toggles fullscreen. Both chords are consumed and never reach the guest.
• Grab ends automatically on focus loss. This is a security property: a grab that survives focus loss would be a keylogger.
• If a session cannot observe focus loss, grab is not offered. The broker logs one line at start saying whether grab is total, partial (a compositor protocol is missing, so some compositor shortcuts still fire) or not offered.
• Hovering gives an absolute pointer; grab gives a relative one. On GNOME the first grab asks once whether the broker may inhibit shortcuts.

Clipboard

Off by default. guest-to-host lets the guest write your clipboard (it can never read it); consent adds host-to-guest only after an explicit paste key. Text only, UTF-8, at most 7168 bytes, rate limited. It also needs QEMU's -chardev qemu-vdagent and spice-vdagent in the guest.

Containers

Bind-mount the socket's directory, not the socket file (a bind-mounted socket inode breaks if the broker restarts). Do not bind the X11 or Wayland sockets or /dev/dri/renderD* for the display-only profile. With rootless podman, --userns=keep-id is the clean answer because the broker checks the peer's uid with SO_PEERCRED.

The guest's cursor

The broker draws the guest's cursor (hover: the host pointer shows the image; grab: an overlay surface), including XOR cursors such as the Windows text I-beam where it can read the frame; where it cannot, it approximates and logs which. Kayfabe composes no cursor itself when the broker advertises this.

Refresh rate

display-max-fps caps the emulated monitor's vblank (24 to 75 Hz; default 75). The broker's --refresh auto tells the guest the host's presentation rate (a 144 Hz host is told 72 Hz), and the guest paces its virtual vblank to it. The kayfabe side falls back to its own clock after 50 ms without ticks.

A Linux desktop on a broker window

scripts/bench/display/interactive.sh is the author's one-command launcher: a Linux guest shown in a broker window on the live desktop, with keyboard, tablet and mouse. It must run as root (QEMU opens /dev/nvidia*, the tap and the GPU lock) while the desktop session is logged in; it stops nothing of the session.

scripts/bench/display/interactive.sh prep      # once: build the broker into /opt/nvkvm-broker; make the guest disk
scripts/bench/display/interactive.sh           # boot; the window appears; Ctrl-C stops it
scripts/bench/display/interactive.sh broker    # only (re)start the broker
scripts/bench/display/interactive.sh stop

What shows, in order: the firmware (TianoCore), the grub menu, the kernel console, then nvidia-drm's console and the Cinnamon desktop on the stock NVIDIA X driver. Environment: KF3_REV, KF_IMG, KF_RAM_MB (8192), KF_FB_MB (8192), KF_SMP (6), KF_EXTRA_QEMU, BROKER_ARGS, KF_BROKER_BACKEND, KF_VNC.

Its QEMU line uses -vga none, a kf3-gpu device with display=on,gop=on,x11-dispsw=on and the broker properties above, plus virtio-keyboard-pci, virtio-tablet-pci,display=kf0,head=0 and virtio-mouse-pci.

What its prep step puts in the guest (an example for your own image): the cinnamon-core, lightdm and X packages; the kernel arguments console=tty1 console=ttyS0 nvidia-drm.modeset=1 nvidia-drm.fbdev=1; nvidia_drm in /etc/modules-load.d; an autologin; and an /etc/X11/xorg.conf that selects Driver "nvidia" for the device at BusID "PCI:0:2:0". Without that xorg.conf, Xorg picked the modesetting driver, whose glamor failed on nvidia-drm, and lightdm restarted X forever [measured 2026-10-08].

Its prep also builds the broker from a pinned revision of the nvkvm-pv repository rather than from broker/ in this tree (KF_NVPV_REV, default badf2d7). If you build broker/ yourself, pass its path with BROKER_BIN.

A Windows desktop on a broker window

scripts/bench/windows/windows_broker.sh shows the Windows 11 guest in a second broker window beside the Linux one (desktop, or run N, and stop, status, broker, reboot, autologon, nvidia). Its default directories (/var/lib/kf-windows-20261005, /opt/nvkvm-broker) belong to the author's host and are overridable (WIN_DIR, BROKER_BIN); read the header before use. The input is PS/2 plus a USB tablet.

For Windows, the recorded way to get a visible desktop is the one in Windows guest: GSP forced on, the three settings, the signed boot-display driver and display=on.

Not documented yet

• A broker package or a service unit.
• A recipe for a broker with display-broker under a sandboxed QEMU (container) end to end.
• Current supported display coverage beyond the named Linux guest driver. The branch codex/sweep-integrate-20261018, not merged, reports earlier AD102 broker runs with Linux 590.48.01 and 595.91.07 [reported in traces/sweep_integrate_20261018/MATRIX.md on that branch, historical revisions 9d6f210ba and 6265b980f]. Those historical runs do not establish current-master display compatibility.
• HDR.

Verified on: the Linux desktop at kayfabe 0e64a960 (host 595.91.07, RTX 4070, GNOME Wayland, 2026-10-08) and the Windows 11 desktop in the README's screenshot (2026-10-21). Not verified end to end: the broker build and the launchers were read from the repository, not run for this page.

Troubleshooting

This page lists symptoms that the repository documents, with the cause and the action it records. It is not exhaustive. Where the repository does not explain a symptom, the page says so.

First rule: if bare metal passes and the guest fails, the defect is kayfabe's. Run the equivalent workload on the host with no VM first (scripts/fastguest/bare_metal_suite.sh, or the program itself); only a bare-metal failure indicts the machine.

Collect evidence before you change anything

• The host's kernel log. A host GPU fault does not raise anything in the guest: the host driver services it, kills the offending channel and prints an NVRM: Xid line in the host's kernel log. Without watching that log, "the copy silently did nothing" and "the GPU faulted" look the same. bash scripts/bench/host_xid_watch.sh <tag> -- <command> runs a command and writes before/after/delta; it exits 6 when a new host Xid appeared across the command and 7 when it could not read the log.
• The guest driver's own dmesg is not in the serial log unless the harness captures it. scripts/bench/boot_capture.sh <tag> stores it, and the host driver's output for that boot, in files.
• Is the job alive? A killed job and a running job look the same if you only check for missing output. Write a start marker and an exit line and check that the process is alive. Note that pgrep -x qemu-system-x86_64 never matches, because the process name is truncated to 15 characters.
• Which binary ran. The device is linked into the QEMU binary, so a source change that was never relinked runs the old device. Print the binary's age, and run the per-revision binary (kf3-bins/<rev>/).

Building

The device refuses to start

The device checks its configuration when QEMU starts and refuses by name rather than clamping silently. If QEMU prints nothing and the serial log is empty, read the QEMU log first: the harness reports FAST_VERDICT=CRASH (the serial log is EMPTY ...), usually because a concurrent run killed it or QEMU failed to start.

A mapping is refused while the guest runs

Linux guest

Windows guest

Reading a TDR on Windows

The Windows application matrix counts these in the guest's System log, for the time since an application started:

• nvlddmkm event 153, the record "Resetting / Restarting TDR", and Display event 4101;
• Windows Error Reporting reports of live kernel events with codes 117, 141 or 116 (VIDEO_TDR_TIMEOUT_DETECTED, VIDEO_ENGINE_TIMEOUT) and bugchecks.

On the kayfabe side, the QEMU log gains a GSP phase Running -> Suspending line for each reset. The two counts are not always equal, so record both: in one recorded run the host side counted 7 resets while the guest log held 3 events and a 0x116 bugcheck.

A cluster of resets can bugcheck the guest: with Windows' default TdrLimitCount=5 within TdrLimitTime=60 s, five resets in a minute became a 0x116 bugcheck. The application matrix therefore moves on to a fresh guest after 3.

Event 153 alone does not establish a TDR: keep the provider, message and timestamp, and correlate them with Display 4101, Windows Error Reporting live-kernel reports or bugchecks, and the QEMU GSP transition log described above [code reading: docs/design/V3_WINDOWS_APP_MATRIX.md sections 2.4 and 2.5]. The investigation separates a harness restart triggered by its 153-inclusive budget from an independently recorded 141 or 4101 TDR.

A TDR does not by itself mean a failure of the application, and the README's rule for a pass is the application's own completion, no host Xid and no GPU reset.

Not documented yet

• A guide to reading every Xid number that can appear. The repository records Xid 31 (MMU fault) and Xid 32.
• Recovery steps after a host GPU channel reset beyond reloading the driver (the provisioning script's GPU_SETTLE does an unload, a PCI reset and a reload, up to three times).
• What a guest-triggered GPU fault does to other VMs sharing the card: not measured.

Verified on: the entries are taken from the scripts, source and records named in each row (kayfabe revisions from afb552ea to 89fc8a222). Not verified end to end: no entry was reproduced for this page.

Security model

Kayfabe has not been security audited. The model below is a design, and the repository's own review of it (a source reading, 2026-10-03) listed gaps. Do not run untrusted or multi-tenant guests on a machine you cannot afford to lose.

Reporting a vulnerability

Report it privately through GitHub's private vulnerability reporting: https://github.com/reindertpelsma/kayfabe/security/advisories/new. Do not open a public issue. Include the revision (the KAYFABE_REV.txt file or the git hash), what you did and what you saw. Reports are read by the maintainer.

In scope: a guest-reachable unbounded read or emit, a guest-controlled host action, or any way for a guest to reach host memory or devices it was not given.

The threat model

• The guest is hostile. Guest userspace and guest root are both untrusted. The boundary inside the guest is also untrusted: unprivileged guest userspace can write any doorbell token, so no check at the trap can establish identity, and the design limits what such a write can reach.
• Host actions are authored, never forwarded. Every call kayfabe makes on the host GPU is built by kayfabe from a few verbs, using the host driver's unprivileged interface. Nothing the guest sends is replayed to the host. Answers to the guest's driver come from facts read from the host GPU through unprivileged controls, from generated family rules, or from values kayfabe fabricates so the driver's own code accepts them.
• The host process is meant to run unprivileged. The host side uses only controls the host driver marks non-privileged and needs no host kernel module. Recorded: one audit ran kayfabe as a throwaway user whose only group was kvm (no capabilities, no sudo). The 30-arm Linux suite passed 30 of 30, the GPU gates 9 of 9, and a Windows 11 guest ran Unigine Heaven in Direct3D 11 at 92.55 fps on average against 93.09 as root (-0.6%, inside a 3.3% run-to-run spread), with no VFIO rebind and no kernel module, driver or udev change [measured, kayfabe 0a746117, RTX 4070, host driver 595.91.07, Ubuntu 26.04, one host]. The record is on the branch claude/rootless-audit-20261016 (commit 32d7319b2, report), not on master. Its qualifiers: it relied on NVIDIA's default mode 0666 for /dev/nvidia*; the host desktop ran on an AMD integrated GPU, so a desktop on the NVIDIA card was not tested; a real window needs the desktop user itself; the CUDA application matrix and the other heavy lanes were not run as that user. Not so yet: the launch scripts in the repository still start as root, and a repeatable rootless test on master is still to be added.
• No CPU executes GPU work, and no completion is forged for work that reached the GPU. Only emulated channels, which have no GPU work behind them, complete inside the VMM.
• Only writes to the device's register window trap. The vCPU never blocks on host work.
• No unsafe code outside three crates and only in files named *_unsafe.rs.

What the isolation is made of

This is the precise version of the home page's "How it is isolated". In short: the VM reaches your GPU through the host's NVIDIA driver, the way an ordinary program does, and the driver and the GPU keep that VM and your other programs apart. The details, each with its source:

• Kayfabe is an ordinary client of the host driver. It opens the device files a CUDA program opens (/dev/nvidiactl, /dev/nvidia0, /dev/nvidia-uvm [measured: strace of a Linux guest run, rootless audit]) and, because it is a QEMU device, /dev/kvm. Every call it makes to the driver is built by kayfabe from a closed list, never replayed from guest bytes, and in NVIDIA's open headers each is marked non-privileged [code reading; how the closed GPU firmware treats a few of them is not verified]. The design document states the property as: every effect a guest can produce on the host GPU is one a local unprivileged process could produce. That is a comparison, not a safety claim.
• The guest never supplies or sees a physical GPU address, and it does use real GPU memory. Its video memory is a real allocation of host VRAM (the store, sized by fb-mb), and its system memory is the hypervisor's shared memory file. The addresses in the guest's GPU page tables are offsets into those two. Kayfabe reads the tables, resolves each offset itself, refuses anything outside the store or guest RAM, and asks the host driver to map its own allocations at the GPU virtual addresses the tables name. So the guest chooses a virtual layout; it does not choose what that layout can reach.
• Guest application work runs on unprivileged host channels, in an address space that holds only that VM's mappings. The command buffers of the guest's applications run unparsed on the real engines. The host address space for such a channel holds only rows derived from the guest's own page tables: no window, no kayfabe memory. Physical addressing needs a privileged channel: on a user-privilege channel the copy engine refused a physical operand with a channel reset, while a virtual operand was delivered [measured, kayfabe 8c084ab7, RTX 4070, driver 595.91.07, 2026-10-08]. For the 3D, compute and video engines this rests on a reading of NVIDIA's open class headers, which expose no physical-operand method; it is not measured.
• Who separates what. Between this VM and your other programs or VMs on the card, the separation is the host driver's and the GPU's: each client has its own address space, and channel privilege decides what a channel may do, as for any two programs on one card. Kayfabe's own responsibility is deciding what goes into the VM's address space. A bug in that address resolution therefore matters more than it would under VFIO, and it is the code that gets hostile-input tests and review. Kayfabe's own GPU programs (the page-table walker and the display copy) run in a CUDA context inside the VMM process, and their bounds checks are a host-memory-safety obligation.
• What is not claimed. No audit, and no statement that a guest cannot reach a bug in NVIDIA's own driver: the host driver is trusted. What a guest-triggered GPU fault does to other VMs on the card is not measured.

Details, including why an IOMMU is not part of the argument, are in the FAQ.

Deployment: one process, and an extra layer you add

Kayfabe is linked into the VMM. There are no helper processes, no IPC and no sandbox of its own.

Confining that process is defence in depth, and it is recommended. The GPU-level separation described above does not rest on your confinement. When kayfabe runs as an ordinary user, it is whatever the host driver gives any unprivileged client. When it runs as root with CAP_SYS_ADMIN, as the launch scripts do today, it holds only because kayfabe never asks for a privileged operation (reviewed in code, not enforced), so the unprivileged way is the better one. On top of that, run kayfabe as an ordinary user and, if you can, in a container or sandbox (systemd-nspawn, bubblewrap, or Docker with /dev/kvm and the NVIDIA device nodes passed in; see FAQ). By design, a compromised kayfabe then has the reach of an ordinary GPU-container tenant on the host driver, plus access to /dev/kvm. A guest first has to break kayfabe to get there, and the path through kayfabe's own authored calls is a closed list of calls an unprivileged process may issue. NVIDIA driver bugs and KVM bugs still apply, and none of this is audited. What an attacker can do after taking over the VMM is outside the security model's scope. The FAQ explains why this matters.

The deployment assumptions the repository's security review lists (as recommendations; it was written to be stated in the support notes) are:

• a confined VMM (recommended as an extra layer);
• one uid per VM;
• CAP_SYS_ADMIN dropped by the launcher (recommended, not enforced);
• host scrub-on-free left enabled;
• profiling restricted to admin users (the host driver's default);
• no --privileged container;
• a PCI topology where the guest cannot unplug the device, until that issue is closed.

This page adds one recommendation of its own: enable the host IOMMU when the host runs untrusted guests. It is not a precondition. It limits a wrong DMA address from a buggy or compromised device to pages already mapped for it, and the argument above does not rely on it (see the FAQ IOMMU answer).

The display broker adds a separate small program so a sandboxed VMM need not touch your display server; see Display and the broker. Its checks include a peer-credential check on the socket and a grab that ends on focus loss.

Harness secrets: the Windows harness generates its password, ssh key and TPM state into a private directory (mode 0700/0600). The TPM state is the secret; never copy or commit it.

What a tenant can still do to others (accepted)

• Completion interrupts are GPU-wide. A tenant that runs GPU work asking for interrupts makes other VMs that registered for the same engine's events take extra, spurious interrupts (a minor denial of service). A guest can tell, from when its interrupts arrive, that some tenant used that kind of engine, and roughly how often. It does not learn which tenant, what the work was, or any data. Any unprivileged host process can already observe the same [measured 2026-10-08, RTX 4070]. The project accepted this, since dropping an interrupt would be a correctness bug. An optional pacing setting, KF3_PT_NSI_MIN_INTERVAL_US (off by default), can cap how often one guest vector is raised.
• Not measured: what a guest-triggered GPU fault does to other VMs sharing the card.
• Resources not bounded per VM at the 2026-10-03 source review: NVENC sessions, runlist time, host BAR1 per VM, GPU time, host RM call rate and host log volume; the page-table walker's work in one refresh is shared across the spaces in a batch, so one guest's layout can make that refresh fail for others. The review's answer for the shared-GPU case is one guest per GPU. These findings are from a source reading dated 2026-10-03; some have since been addressed and the review has not been repeated.

What the repository does not claim

• No independent audit, no penetration test and no formal proof.
• No statement that a guest cannot reach a host-side bug in NVIDIA's driver. The host driver is trusted.
• No claim for a multi-tenant host. The README says the model is designed but not audited, and no public binary ships.

Related

Technical FAQ for how the design works; FAQ for the IOMMU and interrupt questions.

Verified on: the threat model is from the repository's documents of 2026-10-03 (source review at master 7c9234d2), the README (2026-10-21) and the FAQ (2026-10-08, kayfabe 8c084ab7, RTX 4070). Not verified end to end: this model was not tested on this page, and the repository states it is not audited.

Technical FAQ

How kayfabe works and why it is built this way, at the level a user or deployer needs. The full design lives in the repository's design documents, which are not part of this manual. Questions about the IOMMU, isolation and interrupts are in the FAQ. Each answer names its source.

What is kayfabe, mechanically?

A QEMU device (kf3-gpu) that appears to the guest as an NVIDIA GPU. The guest's stock driver talks to it the way it talks to a real card. Kayfabe plays the part of the card's GSP, the on-GPU firmware processor that current NVIDIA drivers send their resource-manager commands to. It answers those commands itself, and has the real host GPU do the work behind them. One host process holds all of it: QEMU, with the kayfabe core linked in. Only QEMU is supported today; the core is designed to be independent of the hypervisor, and that independence is a goal (see Goals and plans).

What does the guest actually execute on?

Three kinds of guest channel, three treatments:

There is no CPU executor for GPU work. Completions of work that reached the GPU are never forged.

Why only the GSP path? Which GPUs can that cover?

Turing and newer. Below Turing there is no GSP at all: the guest driver would run its resource manager on the CPU and drive the hardware directly, and there would be no firmware interface to impersonate. On Windows the stock GeForce driver runs its resource manager on the CPU by default [measured: Code 43 without the setting], which is why the Windows guest needs the EnableGpuFirmware setting.

How does the guest's memory get to the host GPU?

• One store. The guest's video memory is one allocation of host VRAM, sized by fb-mb. A guest framebuffer offset is an offset into that allocation. Guest system memory is the hypervisor's shared memfd.
• Mirror, never adopt. The host twin's address space is filled with kayfabe's own host mappings. Kayfabe never points the host at the guest's page tables.
• The diff. A small GPU program (the "walk kernel", committed as PTX) reads the guest's live page tables in place in video memory and emits only the difference from what the host has confirmed. The host applies it as batched maps and unmaps with one TLB invalidate, and acknowledges run by run. A refused map stays a difference and is retried.
• The guest's TLB invalidate completes only after the host mapping is committed (gate 8).

Why does only BAR0 trap? Does the vCPU ever wait on the host?

Only writes to the device's register window (BAR0) trap. BAR1 and BAR2 are never trapped. The vCPU never blocks on host work: privileged writes go into one ordered ring drained by one thread, and a doorbell write sets a bit in a table that worker threads scan. Nothing blocks on a vCPU or under a lock that a vCPU takes.

What does "authored, never forwarded" mean?

Every call kayfabe makes to the host driver is built by kayfabe from named values; none takes a guest-supplied flag word. The host driver's version is measured on the host and decides the layout of those calls; the guest's version, which the guest declares, only decides how kayfabe decodes the guest's requests. A false declaration can at most make that guest's own requests fail.

Where do per-GPU facts come from?

"Derive, never capture." A GPU's facts are read from the host through unprivileged controls; register offsets and class sets are generated from NVIDIA's open driver headers; only a small family table is maintained by hand. A captured per-GPU table is the thing the design exists to avoid.

The generated parts come from NVIDIA's open-gpu-kernel-modules (MIT) and open-gpu-doc; some capability allowlists derive from gVisor's nvproxy. See Credits and notices.

Do I need to patch the host or guest driver or kernel?

No. Nothing is patched. An optional research patch to the host's nvidia-uvm exists to prove unified-memory demand paging; it is not part of the stock path. Optional enhancements (a helper module in the guest, patched guest or host drivers) are planned, and the stock path will not require them.

How does this differ from the alternatives?

From the README's comparison:

What does running as an ordinary process change?

To the host driver, kayfabe is a GPU client that also holds /dev/kvm, so no host kernel patch or custom host driver is needed. By design, a bug in kayfabe is meant to stay inside that one process, several VMs rely on the host driver's multi-process support, and a compromised kayfabe has the reach of an ordinary GPU-container tenant plus /dev/kvm. None of this is audited, and driver and KVM bugs still apply. The reasons, sources and limits are in the FAQ. What has not been measured is on the site's list.

What does the guest see?

The model and architecture stay truthful: the device takes its PCI identity and die facts from the host GPU. The GPU UUID is a per-VM value so that two VMs do not report the same one (gpu-uuid=; as of 2026-10-08 that was code and GPU-free tests only).

Why do the results come from rented, nested hosts?

Most hardware results were measured on rented hosts that are themselves KVM guests, so every kayfabe guest on them is nested; a VM exit costs about 10 to 40 times as much as on bare metal. Ratios are per box.

Why is performance uneven?

It depends on how often an application rings the doorbell. Every doorbell write from the guest is a trapped write, which is a VM exit. Applications that submit large batches run close to host speed, and applications that launch many small kernels pay for each launch. An optional guest doorbell module that would avoid the exit is designed but not built. Performance numbers, with their revisions and boxes, are in the README.

Why is the CUDA walk kernel inside the VMM?

So that the guest's page tables are read where they live, in video memory, by the GPU, instead of being copied and interpreted by a CPU. Kayfabe's own GPU kernels therefore run in a CUDA context inside the VMM, which is why libcuda has to be present on the host, and why the security review calls the walker's bounds the only barrier on that path.

Why a frozen kayfabe-* set of crates next to kf-*?

The kf-* crates are the product. The older kayfabe-* tree is frozen reference code plus the test client (kayfabe-rm-ladder) that the 30-arm suite and the bare-metal baseline use; nothing in kf-* depends on it.

What do the "gates" and "30 arms" mean?

• Gates (kf-gate1 to kf-gate9, plus a tenth probe) play the guest against the real planes on a real GPU with no QEMU: a real copy on kayfabe's own ring completing through a host event; the host address space matching the guest's tables in both page-table formats; translated kernel copies; doorbells driving workers; passthrough on copy and graphics engines; invalidate completing only after the host map is committed; and the walk kernel's diff protocol.
• The 30 arms are 30 checks run by a raw resource-manager client, one guest boot per arm under a hard time budget, with bare metal as the baseline.

What is not decided yet?

Windows sleep/wake, host-side fault containment between VMs, and a public install path. See the status list on the introduction and in Requirements.

Verified on: the descriptions come from the architecture page and the FAQ (master at 2026-10-21, kayfabe 89fc8a222); the gates and suite are recorded for afb552ea (2026-09-30, RTX 3060). Not verified end to end: this page explains design and was not tested.

Goals and plans

These are goals, not promises. Kayfabe is a research project with no release schedule, and the owner can change any of them. Each item says where it comes from and, in one line, what exists today. Where the repository gives no date or no evidence, the item says so.

Hypervisor independence

Goal (the owner's stated goal, 2026-10-27): kayfabe should not depend on one hypervisor. The design keeps the core independent of the hypervisor, and only a thin shim knows the hypervisor it runs under.

Status today: not implemented. Kayfabe is a QEMU device (kf3-gpu), and only QEMU is supported; a second hypervisor needs a shim that does not exist. By design the core reaches the hypervisor only through two traits (Vmm and HostOps) and has no QEMU dependency, and the display broker is a separate program written to be VMM-neutral.

arm64 hosts

Goal: arm64 (aarch64) support later, after x86-64. The owner's ruling of 2026-10-03 says x86-64 ships first and the boot-display driver stays architecture-neutral so nothing x86-specific creeps in. The owner's priority order of 2026-10-22 puts all GPU families first, then arm64 and several GPUs in one guest.

Status today: no run on arm64 is documented. The continuous-integration job only cross-checks that the core builds for aarch64 and that the boot-display driver builds for an aarch64 UEFI target. Hosts are Linux on x86-64 only.

Several GPUs in one guest

Goal: a guest with several kayfabe devices, one per host GPU, including Windows guests, and a way to choose which GPU serves the display. In the owner's priority order of 2026-10-22 it comes after all GPU families work, Blackwell included; that is a priority, not a rule that nothing else may be started.

Status today: one kayfabe device is created per host GPU. A Linux thin guest with two distinct GPUs passed copy-engine checks serially and concurrently [measured in docs/design/V3_MULTI_GPU_AUDIT.md section 7, revision 74fced87, lease 52664396, a rented 8x RTX 3060 box]. The full 30-arm regression was a separate single-device run. Not measured: CUDA inside a two-GPU guest, unified memory across two GPUs, and Windows or the display with several GPUs. Peer-to-peer access between guest GPUs is refused by name and has no decision yet.

Kayfabe on NVIDIA MIG partitions

Goal (the owner's, 2026-10-28): run a kayfabe process bound to one slice of a GPU that NVIDIA's Multi-Instance GPU (MIG) has partitioned. NVIDIA documents that each MIG instance has its own compute and memory, with separate paths through the memory system. By design, a kayfabe process on one slice would be an ordinary client of that slice, and its VM would see only the slice, which would give each VM a hard resource boundary. Time-slicing does not: NVIDIA's GPU Operator documentation says of it that there is "no memory or fault-isolation between replicas". Sharing a GPU by time-slicing is sharing, not isolation, and this page does not call either a guarantee.

Status today: not tested. No MIG-capable GPU has been available to the project. NVIDIA's table of MIG-supported GPUs (Ampere and later) lists data-centre products and RTX PRO products, and no GeForce card appears in it [retrieved 2026-10-28, supported GPUs]. Kayfabe reads per-die facts from the host GPU, so by design a slice would be described by what the host reports for the slice; that has not been run. Of the dies in NVIDIA's table, GA100 and GB10B are refused by name today, and Hopper has never run on hardware. This goal comes after the GPU families in the owner's priority order.

All NVIDIA families from Turing on

Goal: every family from Turing to Blackwell, Hopper included, is a first-class target, not only the one the author owns. Per-die facts are read from the host GPU, and only a small per-family table is kept by hand. Pre-Turing cards cannot work (there is no GSP firmware to stand in for).

Status today: runs are recorded on Turing (TU116), Ampere (GA106, GA104), Ada (AD104, AD106, AD103, AD102) and Blackwell (GB205), mostly with Linux guests. Hopper and datacentre Blackwell were never run on hardware, and two parts are refused by name (GA100, GB10B). See Requirements.

Windows guests and display

Goal: Windows is the end goal of the Mode-2 design (the guest runs NVIDIA's own driver), with the guest's screen shown on the host desktop through the display broker. Windows sleep/wake and more GPU dies are on the list to fix.

Status today: Windows 11 runs a desktop and a set of applications on the author's RTX 4070 with NVIDIA's 580.88 driver and three registry settings. Sleep/wake does not survive, and one Blackwell die (GB206) fails under Windows with Code 43. See Windows guest and Display and the broker.

Driver versions beyond the tested set

Goal: a driver matrix over the same range nvkvm-pv supports (535 to 610), host and guest versions chosen independently, on every family.

Status today: host drivers 570.148.08, 575.57.08, 580.65.06, 580.95.05, 580.159.04 and 595.91.07, and guest drivers 575.57.08, 580.105.08, 580.159.04, 590.48.01, 595.84 and 610.57.04, have passed the recorded checks, on a few boxes each. Hosts older than 570 are untested. Display layouts exist for one Linux guest version, and Windows has run with 580.88 only. The exact lists are on Requirements.

An installer and a stable interface

Goal: an install path that is easier than VFIO (packaging first, then integration with common virtual-machine managers), and a stable interface for the device. The pre-release checks include a security audit before the first public binary.

Status today: none of these exist. There is no installer, no stable interface, no public binary and no security audit. You build from source with the repository's scripts. See Running the QEMU device.

Optional enhancements

Goal: optional cooperation stages on top of the stock path: a helper module in the guest, and patched guest or host drivers, for things such as doorbell writes without a VM exit and demand-paged unified memory. They are planned and optional, and the stock path never requires them.

Status today: design only, nothing in the stock path depends on it; one research patch to the host's nvidia-uvm exists.

Where plans change

The owner's rulings, with their dates, are kept in the repository. This page is rewritten when a ruling changes; an item here is only as current as the date of its source. The status of what works today is on the introduction and Requirements.

Verified on: every item was read from the repository files named in its comments as of master (2026-10-27). Not verified end to end: nothing on this page was run, and the goals have no delivery dates.

Credits and notices

Licence

Kayfabe is dual-licensed: you may use, copy, modify and distribute it under either the Apache License 2.0 or the GNU General Public License version 2 or (at your option) any later version (Apache-2.0 OR GPL-2.0-or-later). The texts are LICENSE-APACHE and LICENSE-GPL; LICENSE lists the exceptions.

Exceptions you should know before building or redistributing:

• Some files are derived from gVisor's nvproxy (Copyright The gVisor Authors, Apache License 2.0). They are available under the Apache License 2.0 only until they are re-derived independently. LICENSE lists them.
• The kf3-gpu device is built inside a QEMU source tree, and QEMU as a whole is GPL-2.0. The device links the crates that contain the gVisor-derived files. A QEMU binary built from this tree is therefore not yet distributable under the GPL. Read LICENSE, exception 2, before redistributing one.
• third_party/ holds git submodules (the Linux kernel, NVIDIA's open-gpu-kernel-modules and gVisor), each under its own licence. They are reference sources: not built, linked or copied into kayfabe.
• The broker/ directory (the display broker) is marked GPL-2.0 OR Apache-2.0, where the GPL option is version 2 only; either option may be chosen.
• A file that carries its own copyright or licence notice is under that notice. Traces, logs and captures contain data produced by third-party software and hardware.

Third-party notices

THIRD_PARTY_NOTICES.md holds the full notices. In short:

Crates used only by tests and developer tools are listed in Cargo.lock and are not linked into a shipped artifact.

NVIDIA's open-gpu-kernel-modules source is the reference the generated tables are checked against, and the version a given table comes from is recorded with it.

Related projects

Other projects attempt something close to kayfabe: a virtual NVIDIA GPU for a virtual machine, so that the VM runs the stock NVIDIA driver. They are credited here for readers who want to compare approaches. Kayfabe does not copy code from them.

• Caraxes (silasmue/Caraxes, core MIT, guest module GPL-2.0-only; a fork of UniStuttgart-IKR/Leandro). It serves a synthetic NVIDIA GPU over vfio-user with a stock guest driver. Its quickstart requires a patched Cloud Hypervisor and keeps the host NVIDIA driver bound. The quickstart's example run sets CARAXES_DMA=register, which needs a copy of the binary with the cap_sys_admin capability set with sudo; the quickstart also uses sudo for the network device and the guest driver's module options. Its host-access section separately describes the device as submitting work through the host driver as an unprivileged client. That description does not make the documented run sudo-free.
• Open-IOV (open-iov.org). A community project that runs stock NVIDIA drivers in VMs without a vGPU licence. Its overview describes the host module, vgsp.ko, binding the GPU and exposing virtual GPUs through the mediated-device framework, with the distribution's NVIDIA module unbound in that mode. Its pages do not say whether the host can keep using the card. The install guide requires root or sudo for loading the module and creating virtual GPUs. The overview and install guide describe different host-driver arrangements: the overview says the distribution module stays unbound, while the install guide also describes an emulated path using the distribution driver. Read both before choosing a setup.

Trademarks

NVIDIA, GeForce, CUDA and the like are trademarks of their owners. Windows is a trademark of Microsoft Corporation. All other names are the property of their owners. They are used here only to say what kayfabe works with. Kayfabe is not affiliated with or endorsed by NVIDIA or Microsoft.

No vendor binaries

Kayfabe ships no vendor binaries: no NVIDIA or Microsoft driver, firmware, image or installer. Where the tests and guides use them, you obtain them from the vendor yourself.

The Windows harness downloads, each pinned by SHA-256: Microsoft's Windows 11 Enterprise LTSC 2024 evaluation image, the virtio-win 0.1.302 drivers (Red Hat / Fedora project), Win32-OpenSSH 10.0.0.0p2 (Microsoft) and NVIDIA's 580.88 Windows driver. The test tooling fetches other vendors' benchmark and application installers. A draft note in the repository records that the terms of some of these are unclear (for example the Windows evaluation image, the free edition of Unigine Heaven and FurMark) and that some (3DMark, Geekbench scripting) may not be fetched automatically. Read each vendor's terms before you fetch.

Your licences

You must comply with the licences of the drivers and operating systems you run with kayfabe. In particular:

• The NVIDIA driver licence applies to the driver you install on the host and in the guest, including its limits on datacentre deployment and on hosting.
• Windows must be licensed for the way you use it.

Kayfabe's own licence does not grant any rights to third-party software.

Acknowledgements

• NVIDIA, for publishing the open GPU kernel modules and documentation from which the interfaces are generated (MIT-licensed; notices in THIRD_PARTY_NOTICES.md).
• The gVisor authors, for nvproxy, from which capability allowlists derive, and for the ioctl formats it documents.
• The nouveau and nova developers and the Linux kernel, read as reference.
• The QEMU project.
• The virtio-win project and Red Hat, for the Windows virtio drivers the harness installs.
• The author's earlier projects nvkvm-pv and the archived nvkvm prototype (credits in archive/nvkvm/CREDITS in the repository), from which the display broker was imported.

Questions about a licence or notice

Open an issue in the repository. Report a suspected vulnerability privately: see Security model.

Verified on: not applicable to behaviour; the texts are taken from LICENSE, THIRD_PARTY_NOTICES.md and docs/NOTICES.md as of master (2026-10-26). This page is not legal advice, and the repository marks its own provenance policy wording as a draft awaiting legal review.

kayfabe FAQ

STATUS: LIVE, 2026-10-28 (rootless, containers, display broker, two-VM, anti-cheat, isolation, ordinary-process and project-comparison answers revised; the IOMMU and interrupt answers are from 2026-10-08). Questions the VM community asks first, answered from the design and the evidence in this repository. A statement carries its evidence or says it is inferred. Where the repository has no verified answer, the answer says not documented yet. Plans and goals are on Goals and plans.

The basics

Is this vGPU or SR-IOV?

No. Kayfabe is a QEMU device that shows the guest an emulated NVIDIA GPU, and the guest's stock NVIDIA driver talks to it. (QEMU is the only hypervisor supported today; independence from one hypervisor is a goal, see Goals and plans and Does it only work with QEMU? below.) There is no hardware partitioning and no vendor vGPU driver or licence server. In the README's comparison, NVIDIA vGPU needs a supported GPU and a licence, while kayfabe's hardware results so far are on consumer cards, Turing to Blackwell. Using NVIDIA's driver is still subject to NVIDIA's own licence: see Credits and notices.

Does the host keep the GPU?

Yes, that is the point of the design. The host driver stays loaded on the card and the host keeps using it; the guest does not own the device. The README's screenshot shows a Windows guest and a Linux desktop using one GPU at the same time. One caveat for today: the repository's provisioning scripts for test machines stop the host's display manager and purge NVIDIA packages, so they are for throwaway machines, not for your desktop. See Running the QEMU device.

Why not VFIO passthrough?

With VFIO one VM owns the whole GPU and the host does not keep it (README comparison). Kayfabe lets the host and the guest use the GPU together. What you give up is hardware-enforced isolation by an IOMMU: kayfabe's boundary is the GPU's own memory management unit plus kayfabe's checks, and it has not been audited (see the IOMMU answer below and Is it safe to run untrusted guests?). VFIO is also the native reference the README measures kayfabe against. Whether VFIO is safe for untrusted guests is itself contested: Microsoft's Discrete Device Assignment documentation says "You should only use Discrete Device Assignment where the tenants of the VMs are trusted", and Xen's SUPPORT.md says it is "generally not safe to use this feature to expose a physical device to completely untrusted guests". Those statements are about Microsoft's and Xen's own platforms. They are not a measured audit of Linux VFIO, and they do not show that kayfabe is safer.

Why not virtio-gpu (Venus, virgl or a native context)?

The repository has no benchmark or feature comparison with them. What it records is the owner's position in a public discussion (issue #1, 2026-09-28 and 2026-09-29), and the project's positioning note of 2026-10-01, which names generic multi-vendor Linux-guest graphics (the virtio-gpu native context case) as not what kayfabe is for. The position: parts of the native-context infrastructure can be useful for a paravirtual NVIDIA driver, but keeping NVIDIA's proprietary userspace still leaves an NVIDIA-specific interface between the guest's user-mode and kernel-mode drivers. Kayfabe already uses the stock NVIDIA PCI/GSP interface as its guest interface, so native contexts would add little at that boundary. Its compatibility target is NVIDIA's normal guest driver pair, including Windows. Because the guest runs the stock NVIDIA driver, nvidia-smi and CUDA are those of the stock driver [measured, README, 2026-10-21].

Why keep NVIDIA's driver instead of using Mesa/NVK?

Mesa/NVK is a useful open graphics stack, and the owner describes it as a serious driver. Kayfabe's compatibility target also includes CUDA and applications that depend on NVIDIA's proprietary graphics and compute stack. Preserving that userspace is part of the goal, alongside preserving the stock kernel driver. The owner sees room for both approaches to develop.

Why does a Windows guest need an emulated GPU instead of a thin shim driver?

Keeping NVIDIA's proprietary Windows user-mode driver with a replacement kernel driver or shim would require reproducing the private interface between that driver pair. A shared resource manager across Linux and Windows does not make the surrounding Windows interface public. Kayfabe keeps both stock drivers and implements the PCI/GSP-facing device contract below them. The owner states that he has not dissected those private payloads for kayfabe, which does not normally depend on their layouts.

Could another VMM frontend use kayfabe, for example through vfio-user?

The owner sees vfio-user as a possible future VMM frontend. It would add communication between processes to a latency-sensitive path, because kayfabe traps BAR and doorbell accesses. The owner expects that to cost latency without suitable hypervisor fast paths or patches; this is an architectural expectation, and that discussion has no comparative benchmark. Today only QEMU is supported, and independence from one hypervisor is a goal.

How is kayfabe different from nvkvm-pv?

The project's history page (written 2026-09-06) describes it this way: nvkvm-pv forwards the guest's driver calls to the host and needs a custom kernel module in the guest, so the guest is Linux and the host and guest driver versions must match. Kayfabe emulates the device instead, so the guest runs NVIDIA's stock kernel driver, the guest driver version is decoupled from the host's by design, and Windows guests are possible. The same page says that if you want GPU forwarding that works today, use nvkvm-pv; kayfabe is a research project.

Is there something similar?

Caraxes and Open-IOV also work toward running stock NVIDIA guest drivers without a vendor vGPU licence. Their documented host requirements differ; the credits describe those requirements and link to their own documentation. No speed or security comparison with either project is recorded here.

Is there an installer or a download?

No. There is no installer, no stable interface and no public binary; you build it from source. A QEMU binary built from the tree is not yet distributable under the GPL. See Building from source and Credits and notices.

Hardware and drivers

Which GPUs work?

Turing and newer is the hard line. Below Turing there is no GSP firmware interface for kayfabe to stand in for, so older cards cannot work. Of the newer families, the repository records runs on Turing (TU116), Ampere (GA106, GA104), Ada (AD104, AD106, AD103, AD102) and Blackwell (GB205), mostly with Linux guests. Windows guests have run on fewer dies, and one Blackwell die (GB206) fails under Windows today. Hopper and datacentre Blackwell are derived from the open driver's source only and were never run on hardware. Consumer GPUs are the tested class; "any NVIDIA GPU" is the goal, not a claim. The per-die table is on Requirements.

Which host driver versions work?

Every recorded run used the open NVIDIA kernel module. Host drivers that passed the repository's checks are 570.148.08, 575.57.08, 580.65.06, 580.95.05, 580.159.04 and 595.91.07. Hosts older than 570 are untested, and the closed host module is supported in principle but untested. The exact checks and revisions are on Requirements.

Does the guest driver have to match the host driver?

No, by design. The host driver version is read from the host and decides how kayfabe talks to the host driver, and the guest's declared version only decides how kayfabe decodes the guest's requests. In the recorded runs, guest drivers 575.57.08, 580.105.08, 580.159.04, 590.48.01, 595.84 and 610.57.04 passed the CUDA ladder against a 580.159.04 host [measured, September 2026, one GA102 box]. A guest version outside the measured set is refused by name. Two limits: the 30-arm suite is graded for 580.x guests only, and the display layouts were derived for Linux guest 580.159.04 only. Windows has been run with NVIDIA's 580.88 driver.

Does the guest need a special driver or kernel module?

No. The guest uses NVIDIA's own installer and driver, unmodified; nothing is patched on the host or in the guest. A Linux guest needs nothing set. A Windows 11 guest needs three registry settings, listed on Windows guest; without the first one the driver stops with Code 43.

Which guest operating systems and Linux kernels work?

Windows 11 Pro (build 26200) and Windows 11 Enterprise LTSC 2024 (build 26100), both with the 580.88 driver, and Linux guests with the stock driver. For Linux the recorded guest is Ubuntu 24.04 with kernel 6.8; other distributions and kernels are not documented yet. Windows sleep/wake does not survive.

Do I need root, or a patched host?

No host patches are required: the stock path uses the host's stock driver. Optional enhancements (a helper module in the guest, patched guest or host drivers) are planned, never required, and none is built into the stock path today; an optional research patch to the host's nvidia-uvm exists, and the four of the 65 recorded CUDA applications that need unified-memory demand paging fail without it.

Root is not required by the design, and one run without root is recorded, but the scripts in the repository still start as root. Recorded: an audit on one host ran kayfabe as a throwaway user whose only group was kvm (no capabilities, no sudo). The 30-arm Linux suite passed 30 of 30 and the GPU gates 9 of 9, and a Windows 11 guest ran Unigine Heaven (Direct3D 11) at 92.55 fps on average against 93.09 as root, with no VFIO rebind and no kernel module, driver or udev change [measured, kayfabe 0a746117, RTX 4070, host driver 595.91.07, one host]. The record is on the branch claude/rootless-audit-20261016 (commit 32d7319b2, report), not on master. Not recorded: the heavy lanes (the CUDA application matrix and similar) as that user, a desktop on the NVIDIA card itself, and a repeatable rootless test on master, which is still to be added. See Security model.

Sharing and performance

How are resources divided between VMs?

Not documented yet as a feature: no per-VM GPU time quota is described. What is described: each kayfabe device has its own video-memory allocation on the host card, sized by the fb-mb property, and the security review of 2026-10-03 lists NVENC sessions, runlist time, host BAR1 per VM and GPU time as not bounded per VM. Its answer for the shared-GPU case is one guest per GPU.

Can several VMs share one GPU?

Two limited records exist. The September audit reports two overlapping Linux thin guests on one RTX 3060, both copy-engine clients reaching ALL ARMS MET [reported in the hardware report, binary revision 4e9aae9b, host driver 580.159.04, one host, sections 4 and 5]. A later Linux-and-Windows run on one RTX 4070 lasted 3 min 20 s with no new host Xid; the Windows device had to be given a 1024 MiB video-memory store to fit beside the Linux one, and that run checked boot and display only [measured, kayfabe a88764b3, 2026-10-08]. These are functional observations. Not measured yet: throughput, fairness, what a guest-triggered GPU fault does to the other VMs, and host BAR1 accounting across separate VMM processes. A repeatable two-VM test is being added; until it exists, do not rely on sharing, and the security review advises one guest per GPU. A design audit also lists several VMs on one GPU as "works as-is (measured, 2 VMs)"; that line is not a workload result.

By design, each VM is a separate ordinary process on the host driver, so how they share the card is the host driver's multi-process behaviour; NVIDIA documents that by default several processes can use one GPU, with a time-sliced scheduler (MPS documentation). NVIDIA's GPU Operator documentation says of time-sliced replicas that there is "no memory or fault-isolation between replicas" (time-slicing), so time-slicing is sharing without a per-VM quota. Hardware partitions (NVIDIA MIG) would give each VM its own slice of memory and compute; that is a goal and has not been tested.

Can one guest use several GPUs?

One kayfabe device is created per host GPU. The recorded two-GPU thin-guest checks ran copy-engine clients serially and concurrently on two distinct RTX 3060s [measured in the multi-GPU report, section 7, revision 74fced87, lease 52664396, one eight-GPU host]. The separate single-device regression passed all 30 arms; the report does not claim a full 30-arm run on each GPU together. CUDA inside a two-GPU guest, unified memory across two GPUs and peer-to-peer access were not measured, and Windows and display coverage remain open. See Goals and plans.

What is the performance cost?

It depends on the workload. Large batches run close to host speed; many small launches pay for each one, because every doorbell write from the guest is a trapped write (a VM exit), and that cost is much higher on the nested rented hosts most results come from. Examples from the README, each with its limits: a matrix multiply in a Linux guest reached 0.997 of host speed on a rented, nested RTX 3060 box (older revisions), while a separate older campaign recorded eager PyTorch decoding at 0.29 of its own host control. These are separate nested RTX 3060 campaigns, not a matched cross-workload comparison. Windows speed is being benchmarked again on current master; results will be posted in the README.

Does CUDA give the same results as the host?

For the 61 of 65 CUDA applications that pass, the recorded outputs equal bare metal [measured, an earlier revision, a rented RTX 3060], and NVENC and NVDEC output was byte-identical to bare metal in the graded lane [measured, RTX 3060]. Four workloads need unified-memory demand paging and fail correctness checks. At least one recorded run printed SUCCESS with an incorrect result; check numerical output and host/guest faults, not just the exit status. Nothing is recorded for a guest with several GPUs.

Do games with anti-cheat work?

Not documented yet. The recorded games and benchmarks are Minecraft (Java and Bedrock), Unigine Heaven and 3DMark; no anti-cheat protected title is recorded. Whether a given anti-cheat accepts a virtual machine is that anti-cheat's decision, and the repository does not test it. The project's own research notes that products such as Vanguard and FACEIT detect and refuse virtual machines. Anti-cheat is a non-goal: when an anti-cheat fights VMs, kayfabe will not be changed to work around it.

Security and trust

Is it safe to run untrusted guests?

Not yet; treat it as research. The design treats the guest as hostile and builds every host action itself from a few unprivileged verbs, but nothing has been independently audited and there is no penetration test. What a guest-triggered GPU fault does to other VMs sharing the card is not measured. Read Security model before deciding: it lists what is trusted, what is accepted and what is unfinished, and how to report a suspected vulnerability privately.

How is the VM kept apart from my other GPU programs?

The VM reaches the GPU through the host's NVIDIA driver as an ordinary client, and the driver and the GPU keep clients apart, as for any two programs on one card. Precisely: kayfabe opens the same device files a CUDA program does (plus /dev/kvm) and makes only calls that NVIDIA's open headers mark non-privileged, each built by kayfabe and never replayed from guest bytes. The guest does use real GPU memory, but it never supplies or sees a physical GPU address: its video memory is one allocation of host VRAM and its system memory is the hypervisor's shared memory file, its GPU page tables hold offsets into those two, and kayfabe resolves them itself and maps them at the virtual addresses the tables name. The guest's applications run on unprivileged host channels, in an address space holding only that VM's mappings. Deciding what goes into that address space is kayfabe's own job, and a bug there matters more than it would under VFIO. All of this is design and code reading plus one hardware check (the copy engine refuses a physical operand on a user channel); none of it is audited. Details and sources: Security model.

Why does it matter that kayfabe is an ordinary, unprivileged GPU process?

To the host driver, kayfabe is an ordinary GPU client that also has /dev/kvm open. It opens the device files a CUDA program opens (/dev/nvidiactl, /dev/nvidia0, /dev/nvidia-uvm) and /dev/kvm [measured: strace of a Linux guest run in the rootless audit, kayfabe 0a746117, 2026-10-16, one host]. No host kernel is patched and no custom host driver is loaded; the guest uses NVIDIA's own installer. That follows from the design, and what is recorded about running without root is in Do I need root, or a patched host? above. There is no installer yet, so "easy to install" is a goal, not a fact.

What this is meant to buy, by design and not audited:

• Several VMs rely on the host driver's own multi-process support. Each VM is a separate ordinary process, so by design the sharing is the host driver's, not logic of kayfabe's own. Two VMs on one GPU are recorded only at boot level so far; a workload and fairness measurement is pending, and nothing here claims that one VM's VRAM use or timing cannot affect another. See Can several VMs share one GPU?.
• A crash or bug in kayfabe is meant to be a crash of that one process, the VMM, and not of the host kernel. What it can do is bounded by what an unprivileged process with /dev/kvm and a GPU client can do, and the host kernel and driver enforce their own memory protection.
• A compromised kayfabe has the reach of an ordinary GPU-container tenant on the host driver, plus access to /dev/kvm. A guest first has to break kayfabe to get there, and the path through kayfabe's own authored calls is a closed list of calls that an unprivileged process may issue. Most GPU container configuration is the choice of which host libraries to bind-mount, and a library grants no access that the device nodes do not already give (the sister project's reading of the NVIDIA container toolkit). The real addition is /dev/kvm, which the same project's FAQ calls designed for unprivileged use.
• It is the boundary a GPU container host already depends on. On GPU container hosts each tenant's container gets its own GPU; Vast, for one, documents exclusive GPU access per container, never shared between users (Vast instances). Containers share the host kernel, so the host's NVIDIA driver is what separates those tenants' CUDA processes [inferred]. Kayfabe asks that driver only for things an unprivileged process may ask for, so it relies on the same boundary. That also means a deployer can run experimental software in a plain GPU container with /dev/kvm passed in, which is a goal that is not tested yet (see Can kayfabe run inside a container?).

What this does not remove: bugs in NVIDIA's driver that an ordinary process can reach (NVIDIA publishes security bulletins for its drivers), bugs in KVM, and bugs in kayfabe's own code, which is reviewed and not audited. On a host that runs untrusted guests, enable the IOMMU: it limits what a wrong DMA address from a compromised or buggy device can reach to pages already mapped for it, and kayfabe's own argument does not depend on it (see the IOMMU answer below).

Does kayfabe need an IOMMU? Is VM isolation weaker than with VFIO?

Short answer: kayfabe's isolation does not depend on an IOMMU, and that is not a weakness in the way it would be for VFIO, because the guest never drives the physical GPU.

Why VFIO needs one. With VFIO the guest's driver programs the real device and can write any bus address into a descriptor or register. The device then DMAs wherever it is told. The IOMMU is the only hardware that stops a malicious guest driver from pointing the GPU at arbitrary host memory, and with one device per VM its domain is a per-VM domain.

What kayfabe does instead.

• The guest talks to an emulated device. A guest-chosen address is an input that kayfabe checks against that VM's own memory or store, then maps through its own tables. Host RM, the trusted driver on the host, builds the GPU page tables from the memory objects kayfabe registered. A guest value never reaches the device as a raw bus address.
• The VM's work runs on host channels that are unprivileged and live in a GPU virtual address space holding only that VM's mappings. The GPU's own memory management unit enforces the address space. Physical addressing needs a privileged channel: [measured, physical-operand oracle at 8c084ab7 (RTX 4070, driver 595.91.07), 2026-10-08] on a USER-privilege channel the copy engine refuses a physical operand, source and destination, with a channel reset, while a virtual operand is delivered; the 3D, compute and video classes expose no physical-operand method in OGKM's class headers (a reading, not a measurement). Details: traces/phys_operand_oracle_20261008/.
• All VMs share one physical GPU, so they share one IOMMU domain for it. An IOMMU cannot tell VM A's DMA from VM B's here, so it could not provide VM-to-VM isolation even if we leaned on it. The isolation between VMs is the per-channel address space.

What the host IOMMU still does. The GPU's DMA on the host goes through the host IOMMU where the host runs in translated mode (it does on the reference host, DMA-FQ). That limits the effect of a wrong bus address, such as a bug in the driver or the hardware, to pages already mapped for that GPU. It protects the host from any process using the GPU, kayfabe included, and it is configuration dependent (many hosts run identity mode). We do not rely on it, and it is not part of the argument.

Would a guest IOMMU (vIOMMU) help? Not for security. The guest kernel works in CPU physical addresses and converts to an IOVA only when it writes a PTE or sends a command to the device; kayfabe converts it back at once. Translation happens at the two ends only and protects almost no code path; the "device" it would guard against is kayfabe itself. It would be a compatibility feature for guests that ask for DMA remapping. Windows 11 does not require one. Cost and design: design/V3_VIOMMU.md.

Is the IOMMU a strong boundary against a malicious device even on bare metal? Only against DMA to pages the driver did not map. Drivers generally trust what the device writes into pages that are mapped for it (rings, message queues, notifiers) and parse it, and GPU drivers are not designed against a hostile device (inferred; Linux's own model treats internal PCI devices as trusted and has a separate bounce-buffer path for untrusted external ones). That is why our own hostile party is the guest and why kayfabe's input handling, bounds checks and authored host actions carry the weight.

What the boundary is made of. Hardware: the GPU's memory management unit and the channel privilege. Software: kayfabe's checks and host RM. So a bug in kayfabe's address resolution matters more than it would under VFIO, which is why that code gets hostile-input tests and review (docs/OWNER_RULINGS.md §V and the review notes in docs/STATUS_AND_HANDOFF.md).

Paravirtual projects in general. Other paravirtual GPU projects (the owner reports virtio-nvgpu says the same of itself; not re-checked here) and nvkvm-pv have no IOMMU-based isolation, and for them it likewise does not decide the boundary: the guest does not own the device.

Can one VM's GPU work wake another VM? Does a VM learn when other tenants use the GPU?

Short answer: yes, a little, on purpose — the same as for processes on one bare-metal GPU. Decided by the owner on 2026-10-08 (docs/OWNER_RULINGS.md §X).

What happens. A "non-stall" interrupt (a completion signal a program asks for, for example after a copy) is raised by the physical GPU and serviced by the host driver. The host driver's notification is GPU-wide: it says "an engine of this kind finished something", not whose work it was, and the host driver itself wakes every client registered on that engine. kayfabe passes the notification on to every VM whose guest registered for that event (a non-stall event its own driver allocated, which kayfabe records), and to no other VM. It never drops one: a lost completion interrupt leaves a program waiting for work that already finished, which is a correctness bug.

What a tenant can do to others (accepted). By running GPU work that asks for interrupts, a tenant makes the other VMs that registered for the same engine's events take extra interrupts. Each is spurious for them: the waiting program checks its own completion value, finds it unchanged and waits again. This is a minor denial of service, accepted for correctness, and it is the host driver's own behaviour between processes. An optional pacing setting (KF3_PT_NSI_MIN_INTERVAL_US, off by default) can cap how often one guest vector is raised; it delays an interrupt, it never drops one.

What a guest learns (accepted). A guest that registered for an engine's events can tell, from when its interrupts arrive, that some tenant used that kind of engine and roughly how often. It does not learn which tenant, what the work was, or any data. Any unprivileged process on the host can already observe this [measured 2026-10-08, RTX 4070, traces/rawclient_ce_interrupt_20261008/bare_tenant_noise_*]: with another VM on the GPU, an idle raw client saw these notifications in every observation window.

What is not shared. Memory, address spaces, channels and results stay per VM; see the IOMMU answer above. Design: design/the_three_channel_kinds.md §1.2.

Running it

Can kayfabe run inside a container?

That is a goal, and it is expected to work with /dev/kvm and the NVIDIA device nodes passed into the container, but it is not tested or documented for kayfabe yet: no recorded kayfabe run is in a container, and there is no container image or guide. Kayfabe is linked into the QEMU process and has no sandbox of its own, so a container is one way to add the extra layer described in the Security model.

What can be said from the repository: in the rootless audit, a Linux guest run opened only /dev/kvm, /dev/nvidiactl, /dev/nvidia0 and /dev/nvidia-uvm, read /proc/driver/nvidia and the card's /sys/bus/pci entries, and needed libcuda loadable on the host; the display broker additionally uses /dev/udmabuf [measured, branch claude/rootless-audit-20261016]. The sister project nvkvm-pv documents the same idea for its CUDA container: docker run --device /dev/kvm --gpus all with no --privileged and no added capabilities, because /dev/kvm is designed for unprivileged use, and because a container boundary outside the VMM means a VMM breakout lands in the container instead of on the host. Those two reasons apply to kayfabe's single QEMU process as well. nvkvm-pv's other container advice (its guest image, its isolate processes, the NVIDIA_DRIVER_CAPABILITIES list it needs for graphics) is specific to nvkvm-pv and has not been checked for kayfabe. The only container advice written for kayfabe is for the display broker: Display and the broker.

Does it only work with QEMU?

Today, yes: kayfabe is a QEMU device (kf3-gpu), and only QEMU is supported. Independence from one hypervisor is a goal: the core is designed to be VMM-agnostic and only a thin shim knows the hypervisor, but no other hypervisor has a shim. See Goals and plans.

Why a display broker instead of QEMU's own window?

Kayfabe's QEMU is built without the GTK and SDL front ends on purpose, and the display broker (a small separate program) is the way to get a desktop window; VNC stays as a second path. The reasons the repository records:

• No lock-in to one hypervisor. The owner's direction of 2026-10-01 is that frames leave through a hypervisor-agnostic broker, not QEMU's SDL or GTK UI. In the broker relay, only about 230 lines in the QEMU device are QEMU-specific; the connection logic, the frame ring and the input policy are VMM-neutral Rust.
• The VMM stays away from your display server. The broker owns the window, the input and the clipboard, and QEMU holds no connection to your display server at all. That matters most when QEMU is confined; it is the extra layer the Security model recommends.
• One display path to maintain. Pointer grab, absolute or relative input, buttons, the wheel and re-sync are decided in one place (kf-broker), and only a thin shim names the VMM, so a second hypervisor adds a shim rather than a second display path.
• The broker owns the cursor and the pacing. It draws the guest's cursor, including true inverting cursors such as the Windows text caret, and kayfabe composes none; the design makes the broker the frame clock (not built yet), with kayfabe's own clock as the fallback after 50 ms without a tick. The display design notes record problems with QEMU's own front ends: SDL and GTK read a guest cursor with different conventions, GTK warps the host pointer under a broker grab, and GTK's window resizing re-modes the guest.
• Frames are copied on the GPU into a frame kayfabe owns. For a compositor on the same NVIDIA GPU the frame is exported as a dma-buf and never crosses to the CPU; the guest cannot change it after the copy. Compositors on another GPU get a copy in host RAM, with shared memory as the last fallback.
• Native screen output without a trip through system memory (the performance reason). By design, when the compositor runs on the same NVIDIA GPU, each finished frame is copied once inside video memory into a frame object kayfabe allocated itself, and the compositor imports it as a dma-buf. No byte of the frame crosses PCIe to the CPU, and the host's own display path puts it on the screen. A Looking Glass-style setup, which its documentation says transfers completed frames through shared memory to a client window and targets a GPU given to the guest by passthrough, so its frames pass through system memory and are shown in a client window. [Measured on the trusted test host, kayfabe 2a20e699, 2026-10-08: with GNOME on Wayland running on the NVIDIA GPU, the GPU-copy rung was offered and the compositor imported it for 169 of the 170 frames sent.] Not measured yet: frame rate or latency against a Looking Glass-style path, and PCIe traffic itself; the no-copy claim rests on the design and on those rung counters. On a compositor on another GPU the frame goes through host RAM, so the benefit is specific to the same-GPU case.

Do I still need licences for NVIDIA's driver and Windows?

Yes. The NVIDIA driver licence applies to the driver on the host and in the guest, including its limits on datacentre deployment and on hosting, and Windows must be licensed for the way you use it. Kayfabe ships no NVIDIA or Microsoft binaries. See Credits and notices.