STAC-25457 Publish and sign the agent and cluster-agent images from GitHub Actions by LouisParkin · Pull Request #454 · StackVista/stackstate-agent

LouisParkin · 2026-08-05T12:57:23Z

Ports the publishing half of pre_release_main_agent_image / pre_release_cluster_agent_image and both merge_docker_manifest_* jobs from GitLab. #446 landed the build-and-verify halves; this adds the push.

Stacked on STAC-25142-agent-lint-unit (#444), like #446 and #448-#452.

What lands

New job	Workflow	Replaces
`publish-agent-image` (amd64, arm64)	build-deb.yml	`pre_release_main_agent_image` (push half)
`merge-agent-manifest`	build-deb.yml	`merge_docker_manifest_main_agent`
`publish-cluster-agent-image` (amd64, arm64)	build-binaries.yml	`pre_release_cluster_agent_image` (push half)
`merge-cluster-agent-manifest`	build-binaries.yml	`merge_docker_manifest_cluster_agent`

Each publish job needs: the existing image-build job; each merge job assembles the multi-arch tag from the two single-arch ones. All four are added to their workflow's cerberus-notify needs list, so a failed publish on the release branch still reaches Slack.

Gating

if: github.event_name == 'push'. The workflow's push: filter only carries the release branch, so the event check is the whole gate — and a workflow_dispatch on an arbitrary branch cannot publish. Same reasoning the existing cerberus-notify gate uses. PRs build and smoke-test images but never push, which is the point on a public repo.

image-pipeline instead of publish_image.sh

Uses the StackVista/image-pipeline composite actions, matching stackstate-process-agent and the STAC-24837 direction for product repos. Over publish_image.sh that buys:

cosign signatures in both v2 and v3 bundle formats, with post-sign verification on the merged manifest
SBOM + max-mode provenance attestations
canonical SUSE Observability OCI labels
an entrypoint ELF-architecture check that fails arch-mismatched images before signing rather than with exec format error on a node
refusal to overwrite an existing tag

Neither Dockerfile declares ARG BASE_IMAGE, so base-name is passed explicitly — both final stages are registry.suse.com/bci/bci-micro. The agent build also needs --build-arg ARCH; the cluster-agent takes none.

Dropped: the `<branch>-<arch>` tag

publish_image.sh pushed both <branch>-<arch> and <short-sha>-<arch>. Only the short SHA is kept, because nothing consumes the branch tag:

helm-charts-internal pins both images to an 8-character SHA (currently 9516cb41 — stackstate-7.78.2 HEAD)
beest receives the tag as AGENT_HASH_UNDER_TEST, which GitLab set from CI_COMMIT_SHORT_SHA
an org-wide code search for stackstate-k8s-agent:master, :stackstate-7*, :$CI_COMMIT_REF_SLUG and the cluster-agent equivalents returns 0 hits, run against a positive control so an empty result is not a false negative

Tag is cut -c1-8 of the commit SHA, matching GitLab's CI_COMMIT_SHORT_SHA and the chart's pin width.

⚠️ Blocked on STAC-25541

These jobs will fail until StackVista/terraform-infra#75 applies. The stackstate+agent robot behind QUAY_USER has no write grant on stackstate-k8s-agent or stackstate-k8s-cluster-agent — neither repo appears in any team block in quay/locals.tf. GitLab publishes them with the separate legacy gitlabci robot, whose access predates Terraform, which is why this gap is invisible until the GitHub lane runs. It is the same failure that blocked process-agent in STAC-25510.

Still out of scope

pre_release_deb (S3 install.sh publish) and sign_deb (GPG key material) remain unported — both are release-lane concerns needing credentials that do not exist on this repo yet.

Validation

actionlint: no new findings (only the pre-existing self-hosted runner-label noise, which is unavoidable without an actionlint.yaml)
zizmor: No findings to report
YAML parses; every needs: reference resolves against a real job in both workflows
Not yet executed end-to-end — the publish path cannot go green before Revert "STAC-3401 blacklisted processes (#71)" #75 applies

Jira: https://stackstate.atlassian.net/browse/STAC-25457

Ports the publishing half of pre_release_main_agent_image and pre_release_cluster_agent_image, plus merge_docker_manifest_main_agent and merge_docker_manifest_cluster_agent, from GitLab to GitHub Actions. PR #446 landed the build-and-verify halves; this adds the push. Each image gets a per-arch publish job that needs: the existing image build job, and a manifest-merge job that assembles the multi-arch tag from the two single-arch ones. Gated on `github.event_name == 'push'` alone: the workflow's push filter only carries the release branch, so the event check is the whole gate, and a workflow_dispatch on an arbitrary branch cannot publish. This is the same reasoning the cerberus-notify gate already uses. Uses the StackVista/image-pipeline composite actions rather than an open-coded docker push, matching stackstate-process-agent and the STAC-24837 direction for product repos. Over publish_image.sh that adds cosign signatures in both v2 and v3 bundle formats, SBOM and max-mode provenance attestations, canonical SUSE Observability OCI labels, an entrypoint ELF-architecture check that catches arch-mismatched images before they are signed, and refusal to overwrite an existing tag. Neither Dockerfile declares `ARG BASE_IMAGE`, so base-name is passed explicitly; both final stages are registry.suse.com/bci/bci-micro. Tag is the 8-character short SHA, matching GitLab's CI_COMMIT_SHORT_SHA. The `<branch>-<arch>` tag publish_image.sh also pushed is deliberately dropped: helm-charts-internal pins the agent and cluster-agent images to an 8-character SHA (currently 9516cb4, the stackstate-7.78.2 HEAD), and beest receives the tag as AGENT_HASH_UNDER_TEST, so nothing consumes a branch-name tag. An org-wide code search for stackstate-k8s-agent:master, :stackstate-7*, :$CI_COMMIT_REF_SLUG and the cluster-agent equivalents returns no hits, verified against a positive control so an empty result is not a false negative. Both publish jobs need id-token: write for keyless cosign signing, and are added to each workflow's cerberus-notify needs list so a failed publish on the release branch still reaches Slack. Requires STAC-25541 (terraform-infra #75): the stackstate+agent robot behind QUAY_USER has no write grant on stackstate-k8s-agent or stackstate-k8s-cluster-agent, since GitLab publishes them with the legacy gitlabci robot instead. Without it these jobs fail exactly as process-agent did in STAC-25510. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

LouisLotter · 2026-08-06T09:20:14Z

+            [{"registry": "${{ vars.REGISTRY_HOST }}", "username": "${{ vars.REGISTRY_USER }}", "password": "${{ secrets.REGISTRY_PASSWORD }}"}]
+
+      - name: Build, publish, and sign architecture image
+        uses: StackVista/image-pipeline/.github/actions/push-single-arch@6284a6fc006a7cc46a7f00d02c50d5f21b117b63


[P1] Gate publication on image scanning. push-single-arch builds, pushes, attests and signs but does not run scan-image. Both customer images can therefore be published without the mandatory VEX-aware Trivy/Grype vulnerability gate and separate fail-closed secret scan. Please scan the local amd64/arm64 images in mode: gate and make both publish jobs depend on successful scans; the cluster-agent path has the same gap.

LouisLotter · 2026-08-06T09:20:15Z

+  publish-agent-image:
+    name: Publish and sign agent image (${{ matrix.arch }})
+    if: github.event_name == 'push'
+    needs: build-agent-image


[P1] Wait for DEB branding verification before publishing. This job only needs build-agent-image, while test-deb-renaming runs independently. The image can therefore be pushed, signed and made immutable before a failed branding test is known. Add test-deb-renaming to needs (along with the scan gate) so a failing package cannot be published.

LouisLotter · 2026-08-06T09:20:16Z

+          description: Node agent collecting metrics, logs, traces and topology for SUSE Observability.
+          component: stackstate-k8s-agent
+          dockerfile: Dockerfiles/agent/Dockerfile
+          base-name: registry.suse.com/bci/bci-micro:latest


[P2] Pull BCI bases through the SUSE Private Registry proxy. These jobs run on self-hosted docker-public and arm64-xlarge-public, but both Dockerfiles pull BCI stages directly from registry.suse.com. Parameterize those bases and pass registry.tooling.stackstate.io/suse/... to both builds; source-registry-credentials does not rewrite the FROM references.

@LouisParkin some of this feedback is recurring now. Something I find useful is to tell the agent to not only fix feedback like this but also understand why this mistake happened and write down guidance so it does not make the same mistake again.

…roxy BCI bases Address review feedback on #454. Gate publication on the image scan: build-agent-image and build-cluster-agent-image now run image-pipeline's scan-image in `mode: gate` at the house severity (UNKNOWN..CRITICAL, Grype enabled) directly after the smoke test. The publish jobs already depend on the build jobs, so a failing scan now blocks publication instead of only informing. Because build-agent-image also runs for same-repo pull requests, the gate executes on this PR's own CI. Make publication depend on all verification jobs: publish-agent-image now needs test-deb-renaming as well as build-agent-image, so a .deb that still carries DataDog branding cannot reach quay.io. test-deb-renaming lives in this workflow only, so the cluster-agent lane is unaffected. Pull the BCI bases through the SUSE proxy: both Dockerfiles take BCI_IMAGE_REGISTRY as a build argument, defaulting to registry.tooling.stackstate.io/suse/bci. This follows the existing BASE_IMAGE_REGISTRY convention in the same files, so every build path picks it up -- the GitHub workflows, the GitLab jobs, and build_images.sh -- without each caller having to pass a build argument. The apply-oci-labels base-name input stays on registry.suse.com so the provenance label keeps the upstream identity that VEX data is keyed on. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

LouisParkin mentioned this pull request Aug 5, 2026

STAC-25500 Port the DEB signing and pre-release publishing lane to GitHub Actions #455

Open

LouisLotter reviewed Aug 6, 2026

View reviewed changes

Navigation Menu

Uh oh!

STAC-25457 Publish and sign the agent and cluster-agent images from GitHub Actions - #454

STAC-25457 Publish and sign the agent and cluster-agent images from GitHub Actions#454
LouisParkin wants to merge 2 commits into
STAC-25142-agent-lint-unitfrom
STAC-25457-publish-lane

LouisParkin commented Aug 5, 2026

Uh oh!

LouisLotter Aug 6, 2026

Uh oh!

LouisParkin Aug 6, 2026

Uh oh!

LouisLotter Aug 6, 2026

Uh oh!

LouisParkin Aug 6, 2026

Uh oh!

LouisLotter Aug 6, 2026

Uh oh!

LouisLotter Aug 6, 2026

Uh oh!

LouisParkin Aug 6, 2026

Uh oh!

LouisParkin Aug 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

LouisParkin commented Aug 5, 2026

What lands

Gating

image-pipeline instead of publish_image.sh

Dropped: the <branch>-<arch> tag

⚠️ Blocked on STAC-25541

Still out of scope

Validation

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Dropped: the `<branch>-<arch>` tag