Skip to content

Throwaway: full CI test of the one aarch64 image (#258) on linux-omarchy-n1x - #264

Closed
emirb wants to merge 41 commits into
n1xfrom
ci/one-arm-image-full-test
Closed

emirb wants to merge 41 commits into
n1xfrom
ci/one-arm-image-full-test

Conversation

@emirb

@emirb emirb commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Throwaway, not for merging. #258's branch plus what it takes to test it in CI. #258's own workflow never ran: it only triggers on a branch named arm64-generic.

Result

The one aarch64 image, on linux-omarchy-n1x, builds, boots live, installs unattended and boots what it installed. Timings are in the comment below, updated by each run.

What this adds to #258

  • The N1x packages. linux-omarchy-n1x is on no channel yet. test/arm64-n1x-bundle takes the builds omarchy-pkgs' CI already made of its n1x branch and hands them to --package-dir.
  • QEMU 11. test/arm64-container runs the tests in the Arch Linux ARM image the ISO is built in, as the x86_64 tests do, instead of on Ubuntu's QEMU 8.2.
  • One job on the Apple-backed runner. It has KVM and builds the ISO in about 4 minutes. Runs on pull requests from this repository's branches, never from forks.
  • The live boot test reads the screen. The N1x image names no serial console, so the test waited ten minutes on a system that was up after 21 s. The installer on the screen now counts.
  • A comment with the timings, as the x86_64 test leaves.

Not here yet

x86_64 and aarch64 as one [x86_64, aarch64] matrix, and the same in the nightly.

emirb and others added 25 commits October 10, 2026 12:36
An aarch64 build had to be an N1x build: its own kernel, a package
bundle on disk and local source checkouts. None of that can be built or
booted by CI, and nothing else can boot the result.

--platform generic builds the same image without anything board-specific:
Arch Linux ARM's linux-aarch64, the published edge packages, no bundle
and no NVIDIA stack. It boots machines whose firmware describes their
hardware, which is what a VM is, so it is the image CI can install and
test, and the base a board's kernel and packages go on top of.

linux-aarch64 owns its mkinitcpio preset and names its image /boot/Image,
so the live kernel name, preset and initramfs are made after pacstrap,
inside the image (builder/linux-aarch64-live.sh). The approach is the
dragon branch's customize_airootfs.sh.

The generic image keeps both consoles, serial and panel. The N1x build is
unchanged.
No aarch64 ISO has been built by CI. This builds the generic one natively
on every push to this branch, checks that it carries the boot loader, the
kernel, the live initramfs and the root image, and keeps it as an
artifact for three days. The unit tests run beside it.

The push trigger is for the branch: a manual run needs the workflow on
the default branch first.
The runner's Python ships a "test" package. Discovery from the
repository root, as test/all does it, then looks for test.unit inside
that package and finds nothing.
Thirteen of the installer's tests stage a Node tarball named linux-x64,
and the code they exercise looks for the host's architecture, so they
fail on any aarch64 host. quattro runs them on x86; so does this.
On plain Ubuntu four tests fail for reasons that are the runner's, not the
code's: the keyboard tests need Arch's keymaps behind localectl, and two
package-signing tests meet a Python whose Path.is_file raises on a
directory it may not enter. quattro's unit-test.yml runs ./test/all in an
Arch container with systemd for exactly this; this is that step, plus the
tools the aarch64 build test uses.
The aarch64 build test compares the dev key it wrote with cmp, which the
Arch base image does not carry.
The build downloads 3.3 GB of packages into ~/.cache/omarchy on every run
and prunes what it no longer needs. On Namespace that directory can live
on a cache volume. The step is skipped, not failed, on a runner profile
that has none.
A built ISO says nothing about whether it starts. This boots it the way
a machine would, on one of Namespace's Apple-backed runners, the ARM ones
with KVM: UEFI firmware, GRUB from the ISO, the live kernel and
initramfs, the live root. It passes when the live system offers a login
on its serial port, and keeps the serial log and a picture of the screen
either way.

The machine is virt with gic-version=3, its=off and acpi=off: with the
interrupt translation service on, MSIs are lost under nested KVM there
and virtio disks hang.
The generic aarch64 ISO started its kernel and initramfs and stopped
there: mount: /run/archiso/airootfs: fsconfig() failed: Filesystem uses
"zstd" compression. This is not supported. Arch Linux ARM builds
linux-aarch64's squashfs without zstd.

Of what that kernel reads, lz4 keeps what zstd was chosen for, fast
decompression on the page-fault path. Repacked that way on the runner,
the same ISO offered a login 20 s after power-on. The root image grows
from 4,509 MB to 4,814 MB. Other kernels keep zstd.
An unattended install of the generic ISO stopped at its last step:
"Error: bundled Node.js tarball missing from /opt/packages". The ISO
carries Node for its own architecture; quattro's install/user/mise-work.sh
looks for the x64 one on every architecture. The n1x branch of omarchy
picks by uname -m, which is why the N1x image builds its packages from
--local-source. The generic image now does the same, from the n1x
branches of omarchy and omarchy-pkgs.

The ISO also goes to Namespace's artifact store for the jobs that boot
it. GitHub's store took 3 min 20 s to hand 4.5 GB back to a Namespace
runner; it stays as the fallback and for people.
The whole path with nobody at the keyboard, on a Namespace runner with
KVM: the installer takes its answers from a cidata drive, writes the
disk, and the machine restarts into the installed system, which is then
checked over SSH. The answers are the x86_64 integration harness's own
(build_cidata), changed where the architecture shows: Limine's binary,
the kernel, the mirror.

Checked on a runner by hand before this: the install runs to its last
step and writes a Limine entry and a unified kernel image for
linux-aarch64, and after a reset the firmware starts Limine from the
disk.
An unattended install of the generic aarch64 image did everything and
then stopped at its last check: "Validating boot setup failed: no
installed kernel found in target". The installer finds installed kernels
by the pkgbase file Arch's kernel packages leave next to their modules.
Arch Linux ARM's linux-aarch64 leaves none, though its modules, headers
and Limine's unified kernel image were all in place.

A kernel's module directory without a pkgbase is now named by the kernel
the image was built to install. modules.builtin marks a kernel's own
directory, so a leftover of out-of-tree modules is still not a kernel,
and kernels that do record a pkgbase are found exactly as before.
With the kernel recognised, the same install stopped one check later:
"/boot/EFI/Linux/omarchy_linux-aarch64.efi missing or empty". The
Omarchy settings add thunderbolt to the initramfs. linux-aarch64 has no
such module, mkinitcpio reports a name it cannot find as an error, and
limine-mkinitcpio-hook then builds the unified kernel image and does not
install it: "mkinitcpio failed for kernel 7.2.9-1-aarch64-ARCH,
skipping".

Before the boot image is built, the installer now asks modinfo which of
the listed modules the installed kernel has, and writes a drop-in that
sorts last and takes the others back out. With this on the runner, the
install finished, the machine restarted into the installed system, and
Hyprland was running with no failed unit.
The test waited out its whole timeout on an installer that had stopped
after two minutes, and said nothing about why. It now has a way in to
the live system (the live ISO runs cloud-init, so the cidata drive hands
root a key), fails as soon as the installer's log says it halted, and
prints the end of that log. It also checks that no unit failed, that
Hyprland is running, and that the installed system's packages come from
Omarchy's mirror and nowhere else.
The install test's last failing check is the installed system's
mirrorlist. The Omarchy runtime writes it from
default/pacman/mirrorlist-aarch64, which on n1x names Arch Linux ARM's
own mirror. omarchy's arm64-generic branch is n1x plus that one line
changed to arm-mirror.omarchy.org. The workflow builds from it until the
line lands in n1x.
The aarch64 build test runs ssh-keygen, patch and cmp. quattro's
unit-test.yml installs none of them, so it stopped at 'ssh-keygen:
command not found' on every pull request into n1x. With that fixed,
arm64-iso.yml no longer needs its own copy of the unit job.
* Validate Linux boot entries when Limine disables UKIs

* Provide matching kernel headers in boot validation fixtures
They write an x86_64 Limine binary, and the name checked follows the host, so
the tests failed on an aarch64 machine.
The file permission check used an associative array and grep -P, which bash
3.2 and BSD grep do not have, so an ARM image could not be built from a Mac.
It reads the declared paths with sed and looks each one up with grep -F.
The build asked which board an aarch64 image was for, so each platform would
get an image of its own. It no longer asks. The offline mirror carries every
platform's packages from the lists the runtime ships, the installer runs a copy
of the runtime's omarchy-hw-platform and installs the default set of the
platform the machine is, and the build resolves each of those sets against the
mirror.

The kernel is the one choice left, as --kernel. linux-aarch64 is the default.
linux-omarchy-n1x still comes with --package-dir, since no repository publishes
it, and keeps what only the N1x uses on its image, namely the recovery entry,
the rescue hostname, the console options and --dev-ssh.

Snapdragon laptops get no device tree from their firmware, so GRUB cannot boot
them by itself. Once the live kernel and initramfs exist, the build wraps them
in a UKI that carries a device tree for every supported laptop, whenever the
kernel package ships them, and systemd-stub picks one by the machine's SMBIOS
data. A second archiso patch copies that UKI onto the ISO. GRUB boots it by
default only where SMBIOS says the processor is a Snapdragon, and every other
machine boots the kernel directly. The live image gets ukify for that, and the
firmware tools the installer uses.

A runtime without the platform detector, the Snapdragon hardware setup or a
platform's package list is refused, so the ARM workflow builds against
omarchy's n1x.

Co-authored-by: Jimmy Van Veen <jimmy.van.veen@gmail.com>
…ritten

The GPU, audio and compute DSP firmware of these laptops is signed by the
vendor and ships only with Windows. The configurator copies it off the
Windows partition before a full-disk install removes it, an unattended install
does the same, and the copy travels into the target, where the runtime's
hardware setup installs it. Without Windows on the disk, or off Snapdragon,
nothing happens.

Co-authored-by: Jimmy Van Veen <jimmy.van.veen@gmail.com>
The DSP driver stays out of the live initramfs, where loading it can reset the
USB-C port the install media is on. Without it the live system has no USB
hotplug and no battery reading. On the one laptop this is tested on, a service
loads the driver after checking that the live root is entirely in RAM, with no
mounted block device and no active swap.
The README gets an aarch64 section, and the N1x build commands name the
kernel where they named the platform.
…atform drivers as modules

Arch Linux ARM's kernel builds the X1 clock, pin control, interconnect, PMIC,
PHY and bus drivers in. Omarchy's own aarch64 kernel builds them as modules, so
the live initramfs names them, each one optional. Without them the USB ports,
the keyboard and the panel never come up.

The device tree entry boots with iommu.passthrough=0. That kernel defaults to
passthrough, under which the Yoga Slim 7x's keyboard and touchpad read back
empty descriptors and never appear. On Arch Linux ARM's kernel the parameter
changes nothing.

The live UKI takes the device trees a kernel keeps under its own package name
first, and Arch Linux ARM's shared folder otherwise.
A build on linux-omarchy-n1x needs a directory of packages that no
channel serves yet, and until now only someone who had built them by
hand could make one. omarchy-pkgs' CI builds every one of them for its
n1x branch and keeps each build as an artifact named after the package
directory's tree.

test/arm64-n1x-bundle takes a checkout of omarchy-pkgs, fetches the
build of each of those packages at exactly that tree, and writes the
SHA256SUMS the build asks for.
@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

Install test passed

x86_64 vs quattro aarch64 vs quattro
Result ✅ passed ✅ passed
Build the ISO 201.0 s – 287.0 s –
Install 165.8 s +27.1 s 133.2 s –
First boot 18.7 s +0.3 s 13.4 s –
Everyday boot 16.1 s +0.3 s 12.8 s –

Boots are loader, kernel and userspace, without the VM's firmware.

Install phases, x86_64
Phase This PR quattro
Preparing live environment 0.4 s 0.3 s
Preparing install target 0 s 0 s
Installing Arch + Omarchy 133.2 s 107.1 s
Configuring hibernation 0.7 s 0.6 s
Configuring system 4.6 s 4.4 s
Staging provisioning 0.4 s 0.5 s
Finalizing Limine boot 19.7 s 18.5 s
Finalizing user 6.2 s 6.6 s
Configuring login 0.1 s 0.2 s
Configuring SSH access 0.3 s 0.5 s
Configuring Tailscale 0 s 0 s
Configuring DNS resolver 0 s 0 s
Validating boot setup 0 s 0 s
Creating factory snapshot 0.1 s 0.1 s
Install phases, aarch64
Phase This PR quattro
Preparing live environment 0.1 s –
Preparing install target 0 s –
Installing Arch + Omarchy 114.3 s –
Configuring hibernation 0.1 s –
Configuring system 5.9 s –
Staging provisioning 0.1 s –
Finalizing Limine boot 8.3 s –
Finalizing user 3.9 s –
Configuring login 0.1 s –
Configuring SSH access 0.2 s –
Configuring Tailscale 0 s –
Configuring DNS resolver 0 s –
Validating boot setup 0.1 s –
Creating factory snapshot 0.1 s –

Run · 56437c4

emirb added 4 commits October 13, 2026 16:10
…lt in

The aarch64 tests ran on the QEMU of the runner's distribution: 8.2 on
Ubuntu 24.04, where the x86_64 tests run Arch's current one in an Arch
container. test/arm64-container runs a test in the Arch Linux ARM image
bin/omarchy-iso-make builds in, with Arch's QEMU and Arch's UEFI
firmware for the guest, which Arch Linux ARM does not package.
The workflow ran only on pushes to a branch named arm64-generic, so a
pull request that changed the aarch64 image was never built, booted or
installed by CI.

It now runs on pull requests from this repository, and each of its
three jobs runs once per kernel: Arch Linux ARM's linux-aarch64, and
linux-omarchy-n1x with the packages omarchy-pkgs' CI built for it. The
boot and install jobs run in the container with Arch's QEMU.
…named for what they do

The build ran on an Ampere runner, which has no KVM, and handed the ISO
to two more jobs on Apple-backed runners through an artifact store. The
Apple-backed runners run Linux on Apple Silicon and can build the ISO as
well, so one job now builds it, boots it live and installs it, the way
the x86_64 install test works on its one runner. The upload, the two
downloads and their fallbacks are gone.

The checks said "install" and "unit". They now say which architecture
and kernel they are and what they do.
The x86_64 install test leaves a comment with its timings; the aarch64
one left only a summary on the run's page.

A job after the test jobs writes one comment, updated in place by later
runs: per kernel, whether it passed, how long the ISO took to build, the
live ISO to reach a login, the install by the installer's own clock, the
whole of install, reboot and checks, and the first boot. A failed check
is named under the table.

test/arm64-install now keeps the installer's timing file and the boot
analysis, as the x86_64 test does.
@emirb
emirb force-pushed the ci/one-arm-image-full-test branch from 22e1d2b to 7347be7 Compare October 13, 2026 14:11
emirb added 4 commits October 13, 2026 16:14
The group has settled on one aarch64 kernel, linux-omarchy-n1x, so the
image on Arch Linux ARM's linux-aarch64 is no longer something to keep
working. The kernel matrix goes; the job builds and tests the one
image.
The tests run as root in their container, and QEMU writes its
screenshots readable by their owner only, so the upload stopped on the
first one.
The test waited for a login prompt on the serial port. An image on
linux-omarchy-n1x names no serial console on its kernel command line
(console=tty0 acpi=nospcr, for the laptops' panel), so it boots, shows
the installer after about 20 s, and says nothing on the port: the test
waited ten minutes and failed on a working image.

It now also reads the screen, and takes "Start Install" there as the
system being up.
systemd counts the firmware's time from a clock that, in the VM, has run
since the install began: the comment said the first boot took 9 min 33 s
when the kernel and userspace took under 6 s.
@emirb emirb changed the title Throwaway: full CI test of the one aarch64 image (#258), on both kernels Throwaway: full CI test of the one aarch64 image (#258) on linux-omarchy-n1x Oct 13, 2026
emirb added 2 commits October 13, 2026 18:49
x86_64 and aarch64 each had a workflow and each left its own comment,
with different rows.

The install test is now a matrix over the architecture. Both build the
ISO, install it unattended and boot what they installed, and both leave
the same files, so one comment shows the same four rows for each: build,
install, first boot, everyday boot, next to quattro's numbers.

- aarch64 runs on the Apple-backed runner, on linux-omarchy-n1x. A pull
  request from a fork gets x86_64 on GitHub's runner, as before.
- test/arm64-install reboots the installed system once more, for the
  everyday boot the x86_64 test already measures.
- The comment is written by test/install-test-comment, which the comment
  workflow runs from the default branch. Boot times leave out the
  firmware's share: it is the VM's, and on aarch64 its clock has run
  since the install began.
- The separate ARM64 ISO workflow goes.
@omacom omacom deleted a comment from github-actions Bot Oct 13, 2026
emirb added 2 commits October 13, 2026 18:59
A VM has none of the hardware Omarchy's detector looks for, so the
aarch64 install takes the plain default set and never builds the NVIDIA
module: the path an N1x laptop takes was not tested.

test/arm64-install takes a PLATFORM. Given one, the live system's copy
of the detector is replaced, before the installer asks, by one that
names it, and two checks are added: the installer took that platform's
default set, and for aarch64-n1x the NVIDIA module is built for the
running kernel.

The install test runs it on the ISO the job has already built.
omarchy-pkgs renames linux-omarchy-n1x to linux-omarchy (omarchy-pkgs

It is the kernel for every aarch64 board, so it is what an aarch64
build uses unless told otherwise: --kernel is no longer needed, and
names Arch Linux ARM's linux-aarch64 where someone wants that instead.
The image's file name loses the kernel suffix with it.

What the old name switched on stays as it was: the console settings for
the laptops' panel, the N1x recovery entry, the live preset, and the
rule that the kernel comes from a package bundle until a repository
publishes it.

The package filter loses a case. The N1x list used to name a kernel of
its own, which an image on another kernel had to swap for its own; it
now names linux-omarchy, which the filter already swaps.
emirb added 2 commits October 13, 2026 23:55
An aarch64 build could still be told to use Arch Linux ARM's
linux-aarch64, and carried what that took: a script that made it the
live kernel, its own console settings, an lz4 squashfs because that
kernel cannot read zstd, and a fallback to the place it keeps device
trees.

linux-omarchy is the one kernel for every aarch64 board, so the choice
goes: no --kernel, and the build refuses any other kernel. The image
is built one way.

The installer is unchanged. It still installs whatever kernel the image
records, and its tests keep their fixtures.
The pull request comment already leaves it out; the run's summary
printed systemd's line as it came, and on aarch64 that read "16min
55.811s (firmware) ... = 17min 9.658s" for a boot that took 14 s.

In the aarch64 VM the firmware's share is the runner's uptime: the
guest's timer is not reset when the VM starts. The summary now prints
loader, kernel and userspace, as the comment does.
@emirb
emirb force-pushed the ci/one-arm-image-full-test branch from 589f235 to 25cd613 Compare October 13, 2026 22:29
@emirb

emirb commented Oct 15, 2026

Copy link
Copy Markdown
Contributor Author

Replaced by #277, which brings this matrix to quattro.

@emirb emirb closed this Oct 15, 2026
@emirb
emirb deleted the ci/one-arm-image-full-test branch October 15, 2026 10:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants