Skip to content

Build aarch64 packages on native ARM64 runners - #171

Closed
scottjones wants to merge 5 commits into
omacom:masterfrom
scottjones:aarch64-ci
Closed

scottjones wants to merge 5 commits into
omacom:masterfrom
scottjones:aarch64-ci

Conversation

@scottjones

@scottjones scottjones commented Aug 19, 2026 •

Copy link
Copy Markdown
Contributor

Building aarch64 packages on an x86_64 host requires QEMU. This adds a manually dispatched Build aarch64 Packages workflow on ubuntu-24.04-arm, with a native architecture check and unsigned package artifacts retained for 14 days.

Choose edge, rc, or stable, and optionally provide space-separated recipe names. Package channel and release-ring policy still applies; ordinary packages build on edge. Targeted builds must include any required recipes from this repository; package dependencies are not automatically added to the selected set. A full build has a six-hour timeout. A run that produces no package files fails visibly, and the artifact contains only package archives.

To publish, download and extract the artifact on the repository host into build-output/<mirror>/aarch64, then run:

bin/upload-prebuilt --arch aarch64 --mirror <mirror>

Use the same mirror as the build. This signs, promotes, updates the repository database, and syncs under one release lock.

The workflow complements the scheduled architecture support from #277. It does not enable scheduled ARM publishing; the checked-in PUBLISHED_ARCHES default remains x86_64. The PR changes only the workflow and README. #223 adds reusable workflow calls and a repository artifact to the same workflow file and will need to reconcile its changes after this lands.

Validation:

  • All four repository self-test commands passed on the refreshed branch.
  • Workflow YAML parsed successfully; embedded shell scripts passed bash -n; git diff --check passed. The summary script was exercised with empty output (failure) and a package file (success).
  • Native ARM smoke builds at 3cc6f83 passed: tzupdate on edge and symfony-cli on rc, each with one package built and zero failures. Downloaded both artifacts and verified .PKGINFO reports aarch64 and their executables have ELF machine type AArch64.
  • PR self-tests passed at the same commit.

A full repository build and production publication were not exercised.

The build system already supports aarch64 end to end -- bin/build,
bin/sign, bin/update-repo and bin/sync-repo all take --arch, the
Dockerfile bootstraps an Arch Linux ARM rootfs, and omarchy-keyring is
published for aarch64 so that image can bootstrap -- but nothing runs
it, so pkgs.omarchy.org serves no aarch64 tree.

GitHub's ARM64 runners are free for public repositories, so the build
needs no QEMU. The workflow is dispatch-only and takes an optional
package list, and it stops at uploading artifacts: signing and syncing
need credentials only a maintainer has.
The builder container works as its own uid 1000 user, while a GitHub
runner is uid 1001, and make_dir_writable() chowns the mounted output
directories to the host user. The container then cannot write its
incremental omarchy-build database, pacman -Sy fails to open it, and no
makedepends resolve -- the build dies on the first package.

This is invisible on a workstation, where the developer is uid 1000 too.
@scottjones

Copy link
Copy Markdown
Contributor Author

There are now eight open aarch64 PRs on this repo and we're starting to collide. Posting a map here since this PR is the one several of the others already reference.

How they fit

Layer PR Author
Native ARM64 runner #171 (this one) @scottjones
Reusable repo artifact / omarchy.db publication #223 @birkskyum
Recipes + builder toolchain #240 @riverscn
arch=() flips #195, #197, #244 @oceanapplications, @andrewpatrickflynn
Platform bring-up #221 (Snapdragon fw), #222 (Limine) @birkskyum

These are complementary rather than competing. #223 says so directly — it carries this branch's commits so it can be exercised today, and plans to rebase to only its follow-up commits once this merges. #240 is the mirror image: its docs/aarch64-follow-up.md explicitly defers "native AArch64 production scheduling", states it "does not add an emulated GitHub Actions job", and asks that release acceptance be performed on a native AArch64 builder. That is exactly what this PR adds, and it isn't emulated — ubuntu-24.04-arm, free for public repos.

Where we actually conflict

Until a few minutes ago three PRs flipped arch=() on the same two packages, under three different patch filenames for an identical change:

Package #171 #195 #240
tensaku build-for-aarch64.patch add-aarch64-arch.patch aarch64.patch
tzupdate build-for-aarch64.patch add-aarch64-arch.patch aarch64.patch

I've removed those from this PR (see below), so what remains is #195 and #240, which also overlap on asdcontrol, github-copilot-cli, hyprland-preview-share-picker, omasnap, qmk-hid, symfony-cli and v4l2-relayd.

The diffs agree — arch=('x86_64') → arch=('x86_64' 'aarch64') in every case. It's a naming and ownership problem, not a design disagreement.

Proposed sequencing

  1. Done — I've dropped the tensaku/tzupdate changes from this PR. It's now two commits and two files (.github/workflows/build-aarch64.yml and a README note), +154/-1, with no file in common with Own and build nine ARM-compatible packages #195 or Add native AArch64 package build support #240. The remaining overlap with Publish a reusable aarch64 repository artifact #223 is the workflow itself, which is the dependency rather than a conflict.
  2. Merge this, so Publish a reusable aarch64 repository artifact #223 can rebase down to its own commits.
  3. Let Add native AArch64 package build support #240 own the arch=() flips and settle on its aarch64.patch convention, with Own and build nine ARM-compatible packages #195/1password: build for aarch64 #197/grok-bot: aarch64 + 0.30.0 Linux deb names #244 converging on it.

Happy to reorder if maintainers would rather take #240 first — the point is mainly that these want an order rather than parallel resolution of the same files.

@omarchybot

Copy link
Copy Markdown
Collaborator

Re-reviewed at 60d8452 (previous pass was 913d235). Reviewed by Claude Opus 5 and by Codex at xhigh reasoning as an independent second reviewer; where the two agree below I say so, and Codex's independence is not currently guaranteed, so agreement is worth less than the two findings it contributed on its own.

What changed since the last pass. The branch was rebased rather than extended: 913d235 is no longer an ancestor of 60d8452, the "Build tensaku and tzupdate for aarch64" commit was dropped, and the two remaining commits are byte-identical in content to the ones already reviewed. The whole pkgbuilds/ tree at this head is byte-identical to the merge base — no orphaned .omarchy/patches entry, no PKGBUILD referencing a patch that is no longer on disk, and arch=('x86_64') restored in both tensaku and tzupdate. Codex checked every .patch mention in every PKGBUILD against the files on disk and found no dangling reference. So there is no new code in this delta, and two earlier findings are now moot: the collision with #9 and #195 over the same two arch=() flips is gone, and so is the concern about those patches relying on patch default fuzz to survive an AUR bump. The map in your comment checks out with one correction — this PR and #240 still share README.md.

Still open, and new.

  1. .github/workflows/build-aarch64.yml:14-16 (medium) — the maintainer publication recipe in the header comment does not publish a usable repository. bin/promote-build:172-174 deliberately skips omarchy-build.db* when moving packages into production, and bin/update-repo:34-39 is the only thing that runs build/update-repo.sh, whose repo-add at line 66 is what produces omarchy.db. Following sign → promote → sync as written uploads package files and no database, so pacman clients pointed at the aarch64 tree cannot see any of it. The recipe also omits --mirror, and helpers/paths.sh:11-19 defaults MIRROR=edge, so promoting a stable artifact would read the wrong paths. Adding bin/repo update --arch aarch64 --mirror <mirror> before sync, and threading --mirror through all four, is the fix — but it is your publish flow and we have not run it, so we have left the file alone rather than push a recipe we could not exercise.

  2. .github/workflows/build-aarch64.yml:41-43 (low) — the suggested fallback of driving the build "in batches with the packages input" is not dependency-complete. build/build.sh:440-442 selects only the packages named, build/build.sh:511 counts a dependency only when it is also in the selected set, and build/build.sh:60-61 configures the production [omarchy] repo only when a database already exists — which on a clean runner it does not. pkgbuilds/omarchy/PKGBUILD:32 hard-pins omarchy-settings=${pkgver}, so a batch containing omarchy without omarchy-settings fails at makepkg -s. The unscoped default build is unaffected; it is the batching advice that would not work as described. Codex found both of these; neither was in our first pass.

  3. .github/workflows/build-aarch64.yml:129-136 (low, carried over) — path: logs/ can never match. helpers/paths.sh:11 puts LOG_DIR at logs/, and bin/repo:26,39 is the only thing that creates or writes it; this workflow calls bin/build directly. With if-no-files-found: ignore the step is silent, so a failed build produces no diagnostic artifact at all despite the step suggesting otherwise.

  4. .github/workflows/build-aarch64.yml:69-87 (low, carried over) — the chmod -R 777. The diagnosis is right, and we traced it: helpers/docker-helpers.sh:62-69 runs sudo chown -R $(id -u):$(id -g), which succeeds on a runner with passwordless sudo, leaving the bind-mounted directories owned by uid 1001 while build/Dockerfile ends USER builder at uid 1000; chown does not touch the mode, so opening it first is what lets both write. On the pinned ubuntu-24.04-arm label each job gets a fresh single-tenant VM that is destroyed afterwards, so 0777 on two workspace directories is not a privilege or integrity boundary here — it would be a different matter on a persistent or self-hosted runner, which this workflow does not use. A named POSIX ACL (setfacl -R -m u:1000:rwx -m d:u:1000:rwx) would survive that recursive chown and leave "other" non-writable, if you would rather not have the 777 in the file at all; Codex confirmed the ARM64 runner image ships acl. Worth knowing that all three open aarch64 PRs landed on the same answer independently: Add native AArch64 package build support #240 rewrites make_dir_writable() to chmod -R a+rwX and asserts mode 777 in its own test, and Publish a reusable aarch64 repository artifact #223's copy of this workflow does chmod -R a+rwX build-output pkgs.omarchy.org src. This is not out of line with where the repo is going.

Re-verified clean at this head. No fork can reach this workflow: on: carries only workflow_dispatch — no pull_request, pull_request_target, workflow_call or workflow_run anywhere in the workflow set — dispatch needs repository write access, and a fork running its own copy gets its own secrets, not this repository's. No signing key, publishing credential or write-scoped token is reachable: the job is permissions: contents: read, persist-credentials: false, it never invokes bin/sign, bin/repo or bin/sync-repo, and BASECAMP_CHATBOT_URL is scoped to the failure-notification step alone, matching the pattern already in sync-aur.yml and sync-upstream.yml. No shared-database race: helpers/paths.sh:17-18 makes both output paths architecture- and mirror-specific, nothing here writes the published tree, concurrent dispatches get separate VMs, and the artifact name carries github.run_id. The quoting holds — --package "$PACKAGES" as a single argument is reconstructed identically by the arg loop at bin/build:33-40, and the compgen -G non-match is safely inside an if under set -e. The artifact path matches BUILD_OUTPUT_DIR. Codex agreed on all of this, having been asked the same questions independently.

What was not tested. Nothing was executed. This repository has no test suite, and the only thing that can exercise a GitHub Actions workflow on ubuntu-24.04-arm is GitHub's own runner — so every claim above comes from reading the tree, not from a green run. We parsed the YAML to confirm it is well-formed and that on: resolves to workflow_dispatch alone, and traced the uid and path reasoning through bin/build, helpers/docker-helpers.sh, helpers/paths.sh, build/build.sh and build/Dockerfile. Whether a full unscoped aarch64 build actually completes inside the 360-minute cap is unknown and can only be found out by dispatching it.

Where this sits. Waiting on you for item 1 — a few lines in the header comment. Nothing was pushed to your branch. For the maintainer there is a sequencing question this comment's map understates: #223 does not carry this branch's workflow, it replaces .github/workflows/build-aarch64.yml with its own 145-line rewrite that adds a workflow_call trigger taking pkgs_repository and pkgs_ref inputs. On this one file the two are competing implementations rather than stacked, and #223 as it stands also reintroduces the build-for-aarch64.patch files this PR just removed.

@birkskyum

Copy link
Copy Markdown
Contributor

@scottjones if you can land this one, i can rebase the prs i have that build on top.

The maintainer recipe in the header comment published packages without a
database. bin/promote-build skips omarchy-build.db* when it moves packages
into the published tree, and bin/repo update is the only thing that runs the
repo-add that produces omarchy.db, so sign -> promote -> sync uploaded package
files that no pacman client could resolve. Add the update step, and thread
--mirror through all four: helpers/paths.sh defaults MIRROR to edge, so the
recipe as written would have read the wrong tree for a stable artifact.

The batching suggestion on timeout-minutes was not dependency-complete either.
build/build.sh builds only the named packages and counts a dependency only
when it is also in the selected set, and it configures the production
[omarchy] repo only when a database already exists, which is never true on a
clean runner. A batch containing omarchy without omarchy-settings fails at
makepkg -s on its pinned omarchy-settings=${pkgver}. Say what the input is
actually for rather than offering it as a way to split a full build.

The log upload could never match. LOG_DIR is logs/, written only by bin/repo,
and this workflow calls bin/build directly; build/build.sh writes no log files
at all. With if-no-files-found: ignore the step was silent about it, implying
a diagnostic artifact that never existed. The job log is the diagnostic.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@scottjones

Copy link
Copy Markdown
Contributor Author

Thanks — items 1, 2 and 3 are fixed in 7d50d88. All three were comment-only, so the workflow's behaviour is unchanged; what changed is that the comments no longer describe things that do not work.

1. Publish recipe (medium). Confirmed, and the diagnosis was exactly right. bin/promote-build:86,123,173 skips omarchy-build.db* on every path that moves a file into the published tree, bin/repo update is the only caller of build/update-repo.sh, and its repo-add at line 66 is what writes omarchy.db. bin/sync-repo:235-237 then uploads --include "omarchy.*" out of $REPO_DIR — so with no update step there is simply no database file there to upload, and the packages land unresolvable. The recipe now reads:

bin/repo sign    --arch aarch64 --mirror <mirror>
bin/repo promote --arch aarch64 --mirror <mirror>
bin/repo update  --arch aarch64 --mirror <mirror>
bin/repo sync    --arch aarch64 --mirror <mirror>

with --mirror threaded through all four for the MIRROR=edge default you flagged, a note on where to unpack the artifact, and a short paragraph explaining why update is not optional so the next person does not trim it back out.

2. Batching advice (low). Confirmed. build/build.sh:511 only counts a dependency when it is also in PACKAGES_TO_BUILD, and build/build.sh:60-61 gates the [omarchy] repo on a database that does not exist on a clean runner, so a batch has to be dependency-closed. pkgbuilds/omarchy/PKGBUILD:32 pins omarchy-settings=${pkgver} and is the obvious way to trip it. The comment no longer offers the input as a way to split a full build; it says what the input is for and what a batch has to satisfy.

3. Log upload (low). Confirmed, and slightly worse than described — not only does this workflow call bin/build directly while logs/ is written only by bin/repo:26,39, but build/build.sh writes no log file anywhere; it is all stdout. There was nothing for the step to collect under any arrangement, so I removed it rather than repointing it. The job log is the diagnostic.

4. chmod -R 777 (low). Left as is, deliberately. Your own analysis is the reason: ubuntu-24.04-arm gives each job a fresh single-tenant VM that is destroyed afterwards, so this is not a privilege boundary, and the mode has to be opened before make_dir_writable()'s recursive chown for the uid 1000/1001 split to work at all. setfacl would be tidier in the abstract, but #240 rewrites make_dir_writable() to chmod -R a+rwX and #223 does chmod -R a+rwX in its own copy of this workflow — so switching to ACLs here would make this the only one of the three doing something different, for no gain on an ephemeral runner. Happy to revisit if make_dir_writable() gets fixed properly, which would let this step disappear entirely.

On the sequencing note at the end: agreed, and thanks for catching it — I had #223 as stacked on this branch, and it is a competing rewrite of the same file rather than a follow-up. That is worth settling before either merges. cc @birkskyum, since it changes what "rebase on top" means for #223 specifically — #221 and #222 are unaffected.

@birkskyum

Copy link
Copy Markdown
Contributor

@ryanrhughes , this contriubtion has become quite a bottleneck - it's holding back an entire cascade of other PRs related to improving ARM support.

@omarchybot

Copy link
Copy Markdown
Collaborator

Re-reviewed at 7d50d88 (previous pass was 60d8452). Reviewed by Claude Opus 5 and by Codex at xhigh reasoning as an independent second reviewer; Codex's independence is not currently guaranteed, so where it merely agrees below that is worth less than the one confirmation it contributed on its own.

The delta is a single commit. Apart from deleting the log-upload step it is comment text only, so the workflow's behaviour is unchanged except that a step which could never have matched anything no longer runs.

Items 1, 2 and 3 are fixed, and the corrections are accurate. Each claim in the new text was checked against the tree at this head rather than taken on trust.

The publish recipe (lines 12-27). bin/sign:16,21, bin/promote-build:16,21, bin/update-repo:49,54 and bin/sync-repo:29,35 all take --arch and --mirror, and bin/repo:99-130 forwards them unchanged. bin/sign:83 mounts build-output/ whole while build/sign.sh works inside $MIRROR/$ARCH, so build-output/<mirror>/aarch64 is exactly where the artifact has to be unpacked. bin/promote-build:86,123,173 skips omarchy-build.db*; bin/update-repo:39 is the only caller of build/update-repo.sh, whose repo-add at line 66 is what writes omarchy.db. The ordering holds too: bin/promote-build:82-110 refuses to promote unsigned packages, and build/update-repo.sh:12 needs the directory promote creates. Codex contributed the confirmation that settles it — bin/upload-prebuilt:2,29-32 runs sign → promote → update → sync with the same arguments, so the corrected recipe is now identical to the repository's own publishing entry point, which also answers whether clean was missing: it is not in that path either.

The batching note (lines 52-61). Accurate. build/build.sh:439 iterates only the named packages; build/build.sh:508-518 counts a dependency only when it is also in PACKAGES_TO_BUILD and build/build.sh:562-568 installs one only under the same condition; build/build.sh:59-61 adds [omarchy] only when omarchy.db* already exists; pkgbuilds/omarchy/PKGBUILD:32 pins omarchy-settings=${pkgver} and build/build.sh:283 builds with makepkg -scf.

The log upload. Deleting it was right rather than repointing it. logs/ is created only at bin/repo:26 and filled only by the tee at bin/repo:92-140; this workflow calls bin/build directly, and neither bin/build nor build/build.sh opens a log file anywhere.

Item 4, the chmod -R 777. Your reasoning is the same as ours and we are not pursuing it.

One new thing, low, and not a reason to hold the PR. The recipe does not say where it runs. README.md:99-101 says publishing happens on the repository host — "Do not publish from a local checkout instead: only the repository host holds the complete repository and the signing key" — and README.md:219-226 repeats it. That matters for these four commands specifically: build/update-repo.sh:15-16 deletes omarchy.db and omarchy.files and rebuilds them from whatever packages happen to be sitting in that directory, and bin/sync-repo:234-238 then uploads the result over the published database. Followed on a laptop holding a partial aarch64 tree, the recipe would publish a database describing only the locally-present subset. Adding "on the repository host" to the sentence covers it, and bin/upload-prebuilt --arch aarch64 --mirror <mirror> is the one command that already does all four in that order.

What was checked, and what was not. No PKGBUILD changes in this PR at all — git diff 7b6e307..HEAD -- pkgbuilds/ is empty — so there is no pkgver/pkgrel question to answer. bin/check-versions was run on a disposable worker for the record; its output is necessarily identical to master's, since its only inputs are pkgbuilds/ and a published database a fresh checkout does not have. On the same worker the workflow was parsed as YAML: it is well-formed, on: still resolves to workflow_dispatch alone, and the seven remaining steps are the expected ones with the log upload gone. Beyond that nothing was executed: the only thing that can exercise this workflow is GitHub's own ubuntu-24.04-arm runner, so whether a full unscoped build finishes inside the 360-minute cap is still unknown.

Where this sits. Nothing was pushed to your branch and nothing here blocks a merge. For whoever sequences this: since the last pass a fourth aarch64 stack has appeared — #275, #276 and #277 from @maralcbr, which make aarch64 first-class in the scheduled pipeline on the repository host rather than in a dispatched GitHub-hosted job. #277 touches this PR's README.md and overlaps #240 heavily on bin/build, build/Dockerfile and helpers/paths.sh. The competing rewrite of .github/workflows/build-aarch64.yml in #223 is unchanged and still the one collision that has to be decided rather than ordered.

@scottjones

Copy link
Copy Markdown
Contributor Author

@birkskyum This is refreshed against current master and ready for another look. The README conflicts are resolved, rc is available, and the publishing instructions now point to bin/upload-prebuilt on the repository host. I also removed the permission workaround that master now handles and made empty builds fail visibly.

At 3cc6f83, the PR self-tests pass, and both native ARM smoke builds passed: tzupdate on edge and symfony-cli on rc. I downloaded both artifacts and verified the package metadata and executable architecture are AArch64. GitHub reports no merge conflicts.

This still only changes the workflow and README. My preference is to land this first, then rebase #223 so its reusable calls and repository artifact build on the merged workflow. It is ready for maintainer review and merge.

@birkskyum

birkskyum commented Sep 7, 2026 •

Copy link
Copy Markdown
Contributor

Thanks, Scott. Landing #171 first works for me. I've updated #223 to be ready to merge in after this.

@birkskyum

Copy link
Copy Markdown
Contributor

I’ve updated the Dragon ISO to consume the published ARM repositories directly, so this is no longer needed from for snapdragon support.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants