qemu: build the devbox VM as a self-contained repart image by gotha · Pull Request #8 · gotha/dotfiles

gotha · 2026-08-26T07:24:34Z

Replaces run-qemu.sh with nix run .#devbox-qemu, which builds a portable
disk image and boots it.

nix run .#devbox-qemu              # build if needed, then boot
nix run .#devbox-qemu -- --fresh   # discard VM state
nix build .#devbox-qemu            # -> result/devbox_1.raw

run-qemu.sh had drifted: it wanted a raw nixos.img and a prebuilt OVMF, and
its error message pointed at .#packages.x86_64-linux.hala, which no longer
exists.

Why not the stock image path

system.build.images.qemu, images.raw and the systemImage behind
virtualisation.useBootLoader all route through nixos/lib/make-disk-image.nix,
which copies the closure with cptofs (the lkl ext4 tool). On the devbox
closure — 29.2 GiB, ~2500 paths — cptofs stalls:


elapsed	47:26
user CPU	2847 s
kernel CPU	2 s
`rchar`/`wchar`/`syscr`/`syscw`	unchanged over ~6 min
copied	3.0 GiB of 29.2 GiB

Zero syscalls while burning 47 minutes of userspace CPU. It is not slow, it
does not finish. make-disk-image.nix hardcodes cptofs with no way to swap
the copy mechanism, so every stock route is blocked.

virtualisation.useNixStoreImage avoids cptofs (it uses erofs) but is not
portable — nixpkgs documents its store image as built just-in-time at VM start
and "never lands in the store or binary cache", so the host store is still
needed at boot.

systemd-repart writes the filesystem directly and completes in minutes.

Portability

Verified from inside the booted guest:

/dev/vda2   46G  31G  15G  68% /
/dev/vda2   46G  31G  15G  68% /nix/store
0                                          # 9p mounts

The store is a real partition, there are no 9p mounts, and the qemu command
line has no -virtfs. The .raw boots on any host with a UEFI-capable qemu.

Nix database registration

repart.nix copies store paths but never runs nix-store --load-db — this is
what makes the nixpkgs example an "appliance" image that cannot be
nixos-rebuild-ed. Left alone it broke home-manager:

nix-store --realise /nix/store/…-home-manager-generation --add-root …/gcroots/new-home

fails on a path that is physically present but unregistered, activation aborts
before writing ~/.config, and sway comes up with its stock config — no
wallpaper, no dotfiles.

os/linux/repart-image.nix ships the closure's registration file and loads it
in boot.postBootCommands, as sd-image.nix and iso-image.nix do. That runs
in stage-2 init before systemd, so it precedes home-manager-<user>.service. It
also creates /nix/var/nix/profiles/system and /etc/NIXOS, so nixos-rebuild
works inside the guest.

After the fix, in the guest:

check	before	after
`home-manager-gotha.service`	failed	active
`nix-store --realise`	failed	store paths valid
system profile	absent	`system-1-link`
`~/.config/sway/config`	absent	`…-hm_swayconfig`
wallpaper	—	present, file exists in image

Runner details

Backs a writable qcow2 onto the image rather than copying 47 GB.
Checks the overlay's backing file and recreates on mismatch. An overlay from
an older image cannot boot, and one predating this setup has no ESP at all,
which surfaces in OVMF as BdsDxe: failed to load Boot0002 ... Not Found.
Note this discards VM state — unavoidable, since a qcow2 overlay references
blocks of its specific backing image.
DEVBOX_QEMU_DISPLAY and QEMU_OPTS allow headless runs with a serial
console.

Also drops deploy-devbox-qemu, which targeted host devbox.qemu — not in
~/.ssh/config or anywhere in the repo.

Running on Apple silicon

nix run .#devbox-qemu on a Mac boots an ARM build of the same devbox via
-machine virt,accel=hvf -cpu host, so the guest runs at native speed rather
than emulating x86 under TCG. The image comes from distros/devbox-arm, which
filters out everything nixpkgs has no aarch64-linux build for.

Getting that to build and boot on darwin turned up four separate blockers, each
its own commit.

mpd.conf needs a pty the builder does not have

NixOS validates the generated mpd.conf by running mpd under expect(1).
Determinate Nix's external Linux builder gives a build a /dev that is a bare
devtmpfs with no devpts mounted, and builds run unprivileged, so expect dies
with The system has no more ptys and takes the whole system closure with it.
There is no option to skip the check and a derivation cannot mount devpts for
itself, so the images disable mpd — no loss, since no runner attaches a sound
device and the guest has no audio hardware to play to.

repart needs more scratch than the builder's tmpfs

systemd-repart writes the .raw into the build directory, and with
Minimize = "guess" it first stages a full copy of the closure under $TMPDIR
just to measure the smallest partition that fits. Both land on the 3.9 GiB
tmpfs that builder hands a build — half the VM's 8 GiB of RAM — against a
25.4 GiB closure:

Failed to copy '/nix/store/…-gst-plugins-base' to '/build/.#repart…':
No space left on device

The upstream buildPhase pipes through | tee, which swallows that message, so
the build just fails with three blank lines. repartImageOnOut in flake.nix
points both the scratch tree and the image at $out. Result:


`devbox_1.raw`	43,075,702,784 bytes
esp	0.50 GiB, FAT32, `BOOTAA64.EFI` present
root	39.62 GiB, `root-arm64`
buildPhase	41 min 51 s

No-op on a Linux builder, where the build directory is disk-backed anyway.

virt has no PS/2, so the VM had no keyboard

x86 q35 gets a keyboard and mouse free from the emulated i8042; virt gets
nothing, and the command line attached a GPU and no input devices at all. The
greeter never saw a keystroke — the captured cursor was qemu's host-side window
grab, which the guest never sees.

USB HID rather than virtio-input because the guest kernel carries usbcore,
usbhid, hid-generic, xhci-hcd and xhci-pci in modules.builtin, while
virtio_input ships only as a loadable module. usb-tablet reports absolute
coordinates, so the pointer follows the host cursor without a grab.

Adding that controller then broke boot

The EFI variable store records Boot#### entries by device path, so adding a
PCI device shifts enumeration and the stored entry stops resolving. The firmware
falls through to BdsDxe: loading Boot0002 "EFI Internal Shell", which reads as
a dead image. Confirmed by booting the same image twice off throwaway copies,
varying only the varstore:

varstore	result
fresh from `edk2-arm-vars.fd`	boots to `nixos login:`
the one left by the previous run	drops to `Shell>`

The runner now stamps a hash of the qemu command line next to the disk and
discards the varstore when it no longer matches. The hash covers qemu's store
path, so a version bump counts too. The disk — and the guest's state — is left
alone: the ESP carries a fallback \EFI\BOOT\BOOTAA64.EFI, so a blank
varstore boots unaided.

Host key capture

-display cocoa,full-grab=on installs a global event tap so combos like
alt-1 reach sway instead of being eaten by aerospace. DEVBOX_QEMU_DISPLAY
overrides the whole string, so DEVBOX_QEMU_DISPLAY=cocoa turns it back off.

Unverified: macOS only honours the tap once qemu holds Accessibility
permission, which is granted per binary path and so needs re-granting whenever
the qemu store path changes; and whether it beats aerospace depends on which
event tap sits earlier in the chain.

Known issue: virtio-gpu crashes the host qemu

Not fixed in this branch. qemu segfaults on the host while processing a GPU
command:

_platform_memmove            ← EXC_BAD_ACCESS, KERN_INVALID_ADDRESS
iov_to_buf_full
virtio_gpu_simple_process_cmd
virtio_gpu_process_cmdq

41 seconds after launch, on the qemu_main thread. It is a host-side qemu bug,
not the image. bochs-display avoids the code path entirely and binds
bochs-drm cleanly with 1920x1080 available, but exposes no renderD128 render
node — and since sway already runs as --unsupported-gpu, it would likely need
WLR_RENDERER=pixman to start at all. Untested, hence not swapped in.

Setting xres/yres on virtio-gpu was tried and reverted: it pinned the guest
at a fixed 1080p centred in a black frame, losing the dynamic resize that makes
the guest follow the window on fullscreen.

Open points

virtio-gpu segfault above is unresolved; bochs-display is the candidate
fix but needs sway verified on it first.
Minimize = "guess" walks the closure twice — once to measure, once for real
— which is most of the 42-minute build. An explicit SizeMinBytes would
roughly halve it at the cost of a hardcoded size.
The image is 40 GiB; Minimize = "guess" leaves slack. Minimize = "best"
would size it near-exactly, more slowly still.
The closure is large — Zoom, Chromium, CEF, 1Password, two nerd-font
megapacks, rustc, openjdk and qemu itself. A slimmer VM variant would shrink
the image a lot.
bae-qemu still uses images.qemu and will hit the same cptofs stall if its
closure is comparable.
hosts/qemu1/qemu-no-audio.nix is imported directly by mkDevboxQemuImage
rather than living in os/linux/ next to virtio.nix and repart-image.nix.
hosts/qemu1/default.nix itself is not part of the image — repart mkForces
its grub and by-label root away — so the placement is worth a second look.
Both the ssh and gpg keys have to be copied into each fresh VM by hand, and
are lost whenever the overlay is recreated. The gpg key decrypts everything
under secrets/, not just what the devbox reads.

run-qemu.sh wanted a raw nixos.img and a prebuilt OVMF, and pointed at .#packages.x86_64-linux.hala, which has not existed for a while. Replace it with `nix run .#devbox-qemu`, which builds the image and boots it. The image no longer comes from system.build.images.qemu. That, images.raw and the systemImage behind virtualisation.useBootLoader all route through nixos/lib/make-disk-image.nix, which copies the closure with cptofs (the lkl ext4 tool). cptofs spins on the devbox closure - 29.2 GiB, ~2500 paths - and burned 47 minutes of user CPU without a single syscall before being killed. It is not slow, it does not finish. systemd-repart writes the filesystem directly and gets there in minutes. virtualisation.useNixStoreImage was the other candidate and is not portable: nixpkgs documents its store image as built just-in-time at VM start and never cached, so it still needs the host store to boot. The repart image carries its own store on its own partition - no 9p mount, verified - so the .raw boots on any host with a UEFI-capable qemu. repart copies store paths but never populates the Nix database, which is what makes the nixpkgs example an "appliance" image. Left alone it broke home-manager: activation calls nix-store --realise on its own generation, that fails on a path that is physically present but unregistered, and activation aborts before writing ~/.config - so sway came up with its stock config, no wallpaper and no dotfiles. Register the closure on first boot the way the sd-card and ISO images do, which also gives a system profile and /etc/NIXOS so nixos-rebuild works inside the guest. The runner backs a writable qcow2 onto the image rather than copying 47G, and checks the overlay's backing file: an overlay from an older image cannot boot and, if it predates this setup, dies in OVMF as "BdsDxe: failed to load Boot0002 ... Not Found". DEVBOX_QEMU_DISPLAY and QEMU_OPTS make it runnable headless, which is the only way to see the boot at all. Also drop deploy-devbox-qemu: it targeted host devbox.qemu, which resolves nowhere. Entire-Checkpoint: 56c30a00955f

nix run .#devbox-qemu only ever existed for x86_64-linux and hardcoded -enable-kvm, so there was no way to boot a devbox VM from a Mac. Add an aarch64 image and darwin runners for it. distros/devbox-arm is devbox with what nixpkgs cannot build for aarch64-linux taken out. The user package list is filtered on meta.platforms rather than by name - a hardcoded list goes stale the moment someone adds a package to devbox - which drops spotify, slack and zoom-us. Steam has to be named explicitly: no package list mentions it, but its module switches on hardware.graphics.enable32Bit, which asserts on any non-x86_64 system. The two runners differ only in the qemu binary, the firmware and the machine flags, so both now come from one mkDevboxQemuApp template rather than each carrying a copy of the overlay logic. That also replaces install -Dm600, which is a GNU extension the install(1) in macOS base does not have. On Apple silicon -machine virt,accel=hvf -cpu host is real hardware virtualisation, so the ARM guest runs at native speed instead of emulating x86 under TCG - the whole reason for a second image. qemu ships the ARM edk2 blobs itself, both already padded to the 64M the virt machine's pflash wants, so there is no separate OVMF to chase on darwin. virt has no VGA, hence virtio-gpu-pci; there is no GPU passthrough either way, so sway lands on llvmpipe. devbox-qemu-x86 boots the x86_64 image under TCG. It is slow - fine over ssh, rough for sway - and exists because a Mac cannot build a Linux image at all, so the .raw has to be copied in from a Linux builder. Its imageFile is a plain path rather than the store path for that same reason: the store path would make the image a build input of the runner script, and nix run would try to build it and fail before ever reaching qemu. DEVBOX_QEMU_IMAGE overrides it. The backing path is made absolute because qcow2 resolves a relative one against the qcow2's own directory, not the working directory. The ARM image also drops crush and codex, and graphite-cli's shell completions. graphite-cli wraps its prebuilt binary in a buildFHSEnv and then runs gt under bubblewrap to generate those completions; bubblewrap needs a user namespace, which qemu-user cannot create, so the step fails whenever the image is built on an emulating x86_64 builder. crush and codex are compiles with nothing cached for aarch64-linux. The x86_64 image derivation is unchanged by all of this.

Two unrelated blockers, both only visible when the image is built through Determinate Nix's external Linux builder rather than on a Linux host. mpd: NixOS validates the generated mpd.conf by running mpd under expect(1), which needs a pty. That builder's /dev is a bare devtmpfs with no devpts mounted and builds run unprivileged, so expect dies with "The system has no more ptys" and takes the whole system closure down with it. The module has no knob to skip the check and a derivation cannot mount devpts for itself, so the only way past it is to not generate an mpd.conf. No loss either way: none of the devbox-qemu runners attach a sound device, so the guest has no audio hardware and mpd's pipewire output has nowhere to play. repart: systemd-repart writes the .raw into the build directory and, with Minimize = "guess", first stages a full copy of the closure under $TMPDIR just to measure the smallest partition that fits. Both land on the 3.9 GiB tmpfs that builder hands a build - half the VM's 8 GiB of RAM - against a 25.4 GiB closure, so it fails with Failed to copy '/nix/store/...' to '/build/.#repart...': No space left on device and even that is swallowed by the `| tee` in the upstream buildPhase, leaving a build that fails with three blank lines. Point both the scratch tree and the image at $out, which is on the real store. No-op on a Linux builder, where the build directory is disk-backed anyway.

The virt machine has no PS/2 controller. x86 q35 gets a keyboard and mouse for free from the emulated i8042, which is why the x86 runner never showed this, but virt gets nothing and the command line attached a GPU and no input devices at all. The greeter never saw a keystroke; the cursor being captured was qemu's host-side window grab, which the guest never sees. USB HID rather than virtio-input because the guest kernel carries usbcore, usbhid, hid-generic, xhci-hcd and xhci-pci in modules.builtin, while virtio_input ships only as a loadable module - built-in means the greeter has a keyboard with no module load in the path. usb-tablet reports absolute coordinates, so the pointer follows the host cursor instead of needing a grab.

The variable store remembers Boot#### entries by device path, so adding a PCI device shifts the enumeration and leaves entries that no longer resolve. The firmware then falls through to BdsDxe: loading Boot0002 "EFI Internal Shell" which reads as "the OS will not boot any more" even though the image is fine - adding the USB controller in the previous commit did exactly that. The runner only refreshed the varstore when it recreated the disk, and the image had not changed, so the stale one survived. Stamp the hashed command line next to the disk and discard the varstore whenever it no longer matches. The hash covers qemu's store path too, so a version bump counts as a change. The disk, and so the guest's state, is left alone: the ESP carries a fallback \EFI\BOOT\BOOT*.EFI, so a blank varstore boots on its own.

virtio-gpu-pci defaults to 1280x800, which is a cramped desktop. xres/yres set the EDID the guest reads, so they pick the mode sway actually comes up in rather than just widening the list on offer; DEVBOX_QEMU_XRES/YRES override. Verified over ssh into the guest: /sys/class/drm reports 1920x1080 first. full-grab installs a global event tap so system combos reach the guest rather than the host. Without it alt-1 is swallowed by aerospace, which binds it to 'workspace 1' in home-manager/aerospace/aerospace.toml, and sway inside the VM never sees the key. Left as part of the DEVBOX_QEMU_DISPLAY default so DEVBOX_QEMU_DISPLAY=cocoa turns the grab back off without editing anything. The grab is unverified: macOS only honours the tap once qemu holds Accessibility permission, which is granted per binary path and so needs re-granting whenever the qemu store path changes, and whether it wins against aerospace depends on which event tap sits earlier in the chain.

Setting xres/yres cost the dynamic resize. Without them cocoa's windowDidResize feeds dpy_set_ui_info into virtio_gpu_ui_info, which rewrites req_state and regenerates the EDID, so the guest follows the window and fills the screen on fullscreen. With them the guest stayed at a fixed 1080p, centred in a black frame - worse than the 1280x800 default it was meant to fix, since that at least grew to fit. Reverts the resolution half of 7af07cc; the full-grab half stays.

The image ships no credentials, so a freshly booted VM can neither clone over ssh nor decrypt any sops secret, and neither failure says what is actually wrong. The zsh plugin clones fail on `Permission denied (publickey)` because home-manager/git rewrites the https URLs in .zshrc to git@github.com:, and sops-nix reports "0 successful groups required, got 0", which reads as a missing key. Write down both, and in particular the part that is easy to lose an hour to: home-manager creates ~/.gnupg at 755 to place its gpg-agent.conf symlink, and gpg will not use a keyring in a world-readable homedir - so after importing the key sops keeps failing with the identical error until the mode is fixed.

gotha marked this pull request as ready for review August 30, 2026 21:38

gotha added 7 commits August 31, 2026 20:48

gotha force-pushed the qemu-vm-repart-image branch from 1f08e0a to b32bae7 Compare August 31, 2026 17:48

gotha force-pushed the qemu-vm-repart-image branch from b32bae7 to 80bb756 Compare August 31, 2026 18:09

gotha merged commit 065bac1 into main Aug 31, 2026

Navigation Menu

qemu: build the devbox VM as a self-contained repart image - #8

qemu: build the devbox VM as a self-contained repart image#8
gotha merged 8 commits into
mainfrom
qemu-vm-repart-image

gotha commented Aug 26, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

gotha commented Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why not the stock image path

Portability

Nix database registration

Runner details

Running on Apple silicon

mpd.conf needs a pty the builder does not have

repart needs more scratch than the builder's tmpfs

virt has no PS/2, so the VM had no keyboard

Adding that controller then broke boot

Host key capture

Known issue: virtio-gpu crashes the host qemu

Open points

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

gotha commented Aug 26, 2026 •

edited

Loading