Skip to content

ci: add on-demand "Publish images" workflow to force-publish tip of main - #69

Merged
imaustink merged 2 commits into
mainfrom
ci/force-publish-images
Sep 29, 2026
Merged

imaustink merged 2 commits into
mainfrom
ci/force-publish-images

Conversation

@k5s-bot

@k5s-bot k5s-bot Bot commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

Context

I merged 3 PRs (#62, #63, #64) back-to-back without their branches being up to date with main. Because our images are tagged by git tree hash (git rev-parse HEAD^{tree}), that combination can leave the tip-of-main image unpublished, which is what this PR gives us a supported way to fix.

Why merging behind main breaks the tag match

ci.yml's build-and-push publishes an image tagged with the PR-head tree. cd.yml (on push to main) resolves the tag as the merge-commit tree and deploys it.

  • When a branch is up to date, the merge is effectively a fast-forward → merge tree == PR-head tree → the prebuilt image matches and CD just deploys it.
  • When a branch is behind main, GitHub writes a real merge commit whose tree differs from the PR-head tree CI built → the tip-of-main tree has no matching image.

That is exactly what happened here:

PR merge commit merge tree PR-head tree match? CD result
#63 c510971 95cb0c4 95cb0c4 ✅ success (deployed prebuilt image)
#64 d951975 9ab15f0 6691cea ❌ success, but build-missing had to rebuild
#62 (tip) 4f090a4 abcefd9 5bfe453 ❌ had to rebuild the tip image

cd.yml's build-missing job backfills the missing image on the push to main, so the tip does self-heal in the normal case. The gap: there is no supported, on-demand way to force a publish when that backfill doesn't happen — e.g. a push's CD run is superseded in the cd-deploy concurrency group (which intentionally drops older queued runs so only the tip ships), or a bad/stale image needs to be overwritten.

Change

Adds .github/workflows/publish-images.yml — a workflow_dispatch job that force-(re)builds and pushes all three images (frontend, api, collab) for a chosen ref, defaulting to the tip of main.

  • Publishing only — it does not deploy. cd.yml still owns rollout to production.
  • Unconditional build + push (unlike build-missing, which skips when the tag exists), so it also overwrites a bad image under the same tag.
  • Matrix, arc-runner-set arm64 runner, tree-hash tag scheme, and Docker Hub login are kept in sync with ci.yml's build-and-push and cd.yml's build-missing.
  • README's Container images section documents the escape hatch.

How to use it (the "in the meantime" force-publish)

After this merges, run Actions → Publish images → Run workflow on main (leave ref blank) to force-publish the three tip-of-main images.

Note: this turn I could not build/push these arm64 images directly — they require the self-hosted arc-runner-set runners and the repo's registry credentials — so this PR delivers the supported mechanism rather than a one-off manual push. At the time of writing, the tip-of-main CD run (#62 → 4f090a4, tree abcefd9) is queued and will build-missing the tip images and deploy them, so the current tip is being published through the normal path as well.

Follow-up (not done here, per the ask)

The durable fix is the repo-settings change you mentioned — enabling "Require branches to be up to date before merging" on the main branch protection rule so a merge can't produce a tree CI never built. Left out of this PR intentionally; this workflow is the interim tool.

Images are tagged by git tree hash. When a PR is merged while its branch is
behind main, GitHub writes a merge commit whose tree differs from the PR-head
tree that ci.yml built, so the tip-of-main image has no matching tag. cd.yml's
build-missing job normally backfills it on the push to main, but there is no
supported way to force a publish on demand when that does not happen (e.g. the
push's CD run was superseded in the cd-deploy concurrency group).

Add .github/workflows/publish-images.yml: a workflow_dispatch job that
force-(re)builds and pushes all three images (frontend, api, collab) for a
chosen ref, defaulting to the tip of main. Publishing only — it does not
deploy; cd.yml still owns rollout to production. The matrix, arm64 runner, tree
-hash tag scheme and registry login are kept in sync with ci.yml and cd.yml.

Document the escape hatch in the README's Container images section.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BcAqhmKSU8rnQHKcRyyhQ6
@k5s-bot

k5s-bot Bot commented Sep 29, 2026

Copy link
Copy Markdown
Contributor Author

🤖 Starting work on this now.

@k5s-bot

k5s-bot Bot commented Sep 29, 2026

Copy link
Copy Markdown
Contributor Author

🤖 Watch live or take over the session here: https://claude.ai/code/session_01AaaPVXthKvyjAufVqtpePX

@k5s-bot k5s-bot Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Solid, well-scoped escape hatch — matrix, runner, tag scheme, and login are faithfully in sync with ci.yml's build-and-push and cd.yml's build-missing, permissions stays least-privilege, and the empty-ref → dispatch-ref fallback in actions/checkout is correct. All CI checks pass. One narrow point on the build step re: actually honoring the "overwrite a bad/stale image" goal; see inline. Not a blocker.

Comment thread .github/workflows/publish-images.yml Outdated
@k5s-bot

k5s-bot Bot commented Sep 29, 2026

Copy link
Copy Markdown
Contributor Author

The review is posted. #69 (review)

Summary

I reviewed PR #69 (ci/force-publish-images), which adds .github/workflows/publish-images.yml — a workflow_dispatch job to force-(re)build and push the three container images for a chosen ref (default: tip of main), plus a README paragraph documenting it.

Verdict: sound, well-scoped PR — approving in spirit with one narrow, non-blocking note.

What I verified:

  • In-sync claim holds. The matrix (frontend/api/collab, dockerfiles, contexts, build-args), arc-runner-set arm64 runner, git rev-parse HEAD^{tree} tag scheme, and Docker Hub login are byte-for-byte consistent with ci.yml's build-and-push and cd.yml's build-missing. The only intended divergence is the unconditional push (vs. build-missing's manifest-inspect skip).
  • Correctness of the default. Empty ref input → actions/checkout@v4 falls back to the dispatch ref (github.sha, i.e. tip of the selected branch), so the comment's claim is accurate. git rev-parse HEAD^{tree} works under the default shallow checkout.
  • Security. permissions: contents: read is least-privilege; the ref input is only passed to checkout's with: (not into a shell), so no injection surface; secrets are the same registry creds used elsewhere.
  • CI: all 9 checks pass (including build-and-push, which republished this PR's tree).
  • YAML parses cleanly.

The one inline comment (with a one-cli…


✅ Opened pull request: imaustink/glyph#69

@k5s-bot

k5s-bot Bot commented Sep 29, 2026

Copy link
Copy Markdown
Contributor Author

🤖 Starting work on this now.

@k5s-bot

k5s-bot Bot commented Sep 29, 2026

Copy link
Copy Markdown
Contributor Author

🤖 Watch live or take over the session here: https://claude.ai/code/session_01Bsb5Etje6GZdt1tCkF7xsS

Address review: the "overwrites a bad or stale image" promise only held
for a corrupt/mis-pushed tag. If the build itself produced the bad image
(a base image that moved under a floating tag, or a poisoned local layer
cache on a reused arc-runner-set runner), a plain docker build reuses
those cached layers and reproduces the identical bad image, so the
re-push fixes nothing. This is the deliberate force path, so bust the
cache and re-pull the base for a genuinely fresh rebuild.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bsb5Etje6GZdt1tCkF7xsS
@k5s-bot

k5s-bot Bot commented Sep 29, 2026

Copy link
Copy Markdown
Contributor Author

Picked this back up from the ai-triage label. Summary of what I changed and why:

Addressed the one review comment (#discussion_r on publish-images.yml:84, now resolved):

  • Changed the force-publish build to docker build --pull --no-cache (commit b47e96f). The PR's own comment promised this path "overwrites a bad or stale image," but a plain docker build reuses cached layers — so if the build itself produced the bad image (a base image that moved under a floating tag, or a poisoned local layer cache on a reused arc-runner-set runner), the re-push would reproduce the identical bad image and fix nothing. --pull --no-cache makes the rebuild genuinely fresh. This intended divergence from ci.yml (which builds a brand-new tree, so cache reuse is fine and desirable) is documented in the step comment.

Branch sync: the branch was already up to date with main (0 commits behind), so no rebase/merge was needed.

Deliberately not changed:

  • The PR description — it still accurately describes the workflow; the fix only strengthens the "overwrite a bad image" behavior it already claimed, so no wording is now inaccurate.
  • The --pull --no-cache flags were not added to ci.yml/cd.yml's builds — they build unique trees where layer caching is a legitimate speedup; only this explicit force path needs the guarantee of a fresh build. Kept in scope.
  • The branch-protection follow-up ("Require branches to be up to date before merging") remains out of scope per the original ask.

All 9 CI checks passed on the prior commit; a fresh CI run is in progress on b47e96f as of this comment.

@k5s-bot

k5s-bot Bot commented Sep 29, 2026

Copy link
Copy Markdown
Contributor Author

Everything is done. CI is re-running on the new commit (the workflow I touched is workflow_dispatch-only, so it doesn't run in PR CI — the passing/pending checks come from ci.yml, which was green on the prior commit and is unaffected by this change).

Summary

I picked up PR #69 ("ci: add on-demand 'Publish images' workflow") from the ai-triage label and got it ready to merge.

Feedback addressed — there was exactly one review comment (from the prior review), on publish-images.yml:84:

  • The reviewer noted the "overwrites a bad or stale image" promise only held for a corrupt/mis-pushed tag. If the build itself produced the bad image (a base image that moved under a floating tag, or a poisoned local layer cache on a reused arc-runner-set runner), a plain docker build reuses those cached layers and reproduces the identical bad image, so the re-push fixes nothing.
  • Fix (b47e96f): changed the force build to docker build --pull --no-cache and expanded the step comment to explain why this force path deliberately differs from ci.yml (which builds a fresh tree where caching is a fine speedup).
  • Replied to the comment and resolved the thread.

Branch sync: already up to date with main (0 commits behind) — no rebase/merge needed.

Deliberately not done: didn't add --pull --no-cache to ci.yml/cd.yml (they build unique trees where caching is legitimate; only the explicit force path needs the guarantee); left the PR …


✅ Opened pull request: imaustink/glyph#69

@k5s-bot k5s-bot Bot removed the ai-triage label Sep 29, 2026
@imaustink
imaustink merged commit e03cbf3 into main Sep 29, 2026
9 checks passed
@imaustink
imaustink deleted the ci/force-publish-images branch September 29, 2026 13:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants