Workflow yaml carries deciding logic inline, where no test can reach it

[usirin]

Pitch

Problem: 66 multi-line run: | blocks and actions/github-script steps across 40 workflow files carry logic no test can reach. The audit on this issue sorted them: 51 decide (parse, branch, retry, call an API), 15 only relay. A wrong pattern inside a yaml string shows up as a red main, not a red test — #9507 was exactly that, three days red from a regex in .github/workflows/deploy.yml. The same logic is also copy-pasted: diff-basis resolution appears inline 5 times while packages/fabrika-cli/src/guard/changed-files.ts is already the tested core for it, the sticky preview-comment writer has 3 inline copies (one already drifted, #9521), and the "file a needs-triage issue" block appears 4 times while fabrika report file does it.
Arc: axis:pipeline-hardening
Appetite: 3 cycles
Rabbit-holes: Moving all 51 at once. publish.yml's tag grammar is read out of the yaml by publish-isolation-guard (ADR 0201), so moving it without re-pointing that guard silently zeroes the gate. Inventing new packages — the founder ruled no ci-cli, no phoenix-ci. Pulling phoenix detail into packages/fabrika-cli, which the portability guard reds.
No-gos: The 15 relay-only blocks and the 7 the audit marked fine inline (pinned binary installs, bot commit, step summary). Any home not yet ruled: the 8 deploy/preview scripts (packages/anka-ops is the audit's suggestion, unruled) and the 3 phoenix-only scripts (publish.yml tag grammar, release dispatch roster, design manifest firewall) stay unmoved until the founder names their home.

Epic — awaiting plan

plan-epic appends its plan and dependency topology below.

Original brief (verbatim)

Summary

Many workflow files hold multi-line inline scripts that parse, branch and retry. None of that logic runs under a test. The #9507 deploy break was one of these: a regex inside a yaml string, red on main for 3 days.

What I was doing

Tracing the red web deploy on main (#9507) and reviewing its fix in PR #9510. The founder then said, on 2026-09-20, that inline scripts in yaml files are not testable and he does not want them.

What I observed

Across 38 files in .github/workflows/ there are 62 multi-line run: | blocks and 4 actions/github-script steps. By file, inline blocks plus github-script steps:

• deploy.yml: 9 + 2
• run-evidence.yml: 7
• release-please.yml: 6
• ci.yml: 5 + 1
• pr-cleanup.yml: 4 + 1
• heal-ci-sweep.yml: 4

In deploy.yml the inline logic includes the worker URL scrape, the prod health verify loop (6 retries with back-off and two fail-closed branches), and a marker-keyed read-modify-write upsert of the preview comment.

PR #9510 fixes the broken scrape but keeps it inline, as a sed one-liner inside the yaml. Its correctness was shown by hand against three real logs in the PR, not by a test that stays in the repo.

The repo's working rule in CLAUDE.md says repository tooling uses Node and Effect CLI with a testable decision core and a thin bin, and that shell relays tool results without owning decisions. These blocks do not follow it.

I did not classify all 62. A block that only calls a bin is fine. The ones that parse, branch, retry or decide are the concern.

Why it matters

Logic in a yaml string is only exercised by a live run, often only after merge, so a wrong pattern shows up as a red main instead of a red test. The next Alchemy output change could break the scrape the same way. How many of the 62 blocks carry real decisions is unknown until someone sorts them.

Pointers

• .github/workflows/deploy.yml (scrape, verify loop, preview comment upsert)
• .github/workflows/ci.yml, run-evidence.yml, release-please.yml, pr-cleanup.yml, heal-ci-sweep.yml
• CLAUDE.md working rules (testable decision core, thin bin)
• Web deploy reds on every main push: the prod health gate finds no worker URL #9507, PR fix: Web deploy reds on every main push: the prod health gate finds no worker URL #9510 (the scrape, still inline)
• A red post-merge run on main reaches nobody: 8 failed web deploys sat unseen for 3 days #9508 (its failure hook is being built with a tested core, per founder ruling on that issue)
• Run 35531984366 job 106134117846 (real Alchemy prod output, usable as a test fixture)

Suggested next step (non-binding)

Sort the run blocks into relay-only and deciding. Move the deciding ones in deploy.yml first, scrape and health verify, into a tested CLI core fed by fixture logs. Consider a guard that reds a new deciding inline block.

---

_{Filed by an agent · session 56fedc1c-913a-4bc4-baa5-2b748ea48b9b · branch main · 2026-09-20T20:07:33Z}

[usirin]

Founder ruling, 2026-09-20: one CLI package

umut ruled that all workflow scripts live inside ONE CLI package, not one small package per job.

• Home: a single phoenix-owned Effect CLI workspace package, packages/ci-cli (@kampus/ci-cli). Grouped verbs, each with a pure tested core and a thin bin the yaml calls.
• Lane A red post-merge run on main reaches nobody: 8 failed web deploys sat unseen for 3 days #9508 creates the package with its first verb, the deploy failure alert.
• Under this issue, the deploy URL scrape and the prod health verify move in next, as verbs in a deploy group, tested against the real Alchemy logs (run 35531984366 job 106134117846 is a usable fixture).
• "All of these scripts" also reaches the single-purpose bins the workflows call today: packages/worker-relevance, packages/migrations-guard, packages/orphan-sweep, packages/preview-seed. Folding them into the one CLI is in scope here as later steps, so expect this to be more than one PR.
• Not packages/fabrika-cli. It ships to other repos and has to stay free of phoenix details; the portability guard reds them there.
• packages/pipeline-cli is a dead directory holding only node_modules. It is not the home.

[usirin]

Audit result, 2026-09-20: what the inline scripts do and where they belong

Four read-only auditors sorted every multi-line run: | block and every actions/github-script step. Row counts matched the expected 66 (62 + 4). These are candidate verdicts from one pass, not rulings.

Kind	Count
Relay only (calls a bin, passes values)	15
Decides (parses, branches, retries, calls an API)	51

Where the 51 deciding scripts would land:

Home	Count	Examples
packages/fabrika-cli (generic delivery pipeline)	33	diff-basis resolution, sticky PR comment upsert, "file a triage issue" blocks, exit-code gates, epic autoclose, changelog gatherer, secret-scan staging, stale merge-ref guard
packages/anka-ops (generic ops for an anka-built app)	8	deploy URL scrape, prod health check, preview D1 lookup, preview flag forcing, prod-destroy refusal, teardown checks for workers and Flagship, await-preview probe
phoenix only	3	publish tag grammar, release dispatch roster, design manifest firewall
Fine to stay inline	7	pinned binary installs, bot commit, step summary

Duplication the auditors found:

• Diff-basis resolution is written inline 5 times (ci.yml twice, deploy.yml, gitleaks.yml, leak-guard.yml), and packages/fabrika-cli/src/guard/changed-files.ts is already the tested core for it.
• The sticky preview comment has 3 inline writers; fabrika already has a tested reader (src/capture/resolve.ts) and no writer. One copy drifted: pr-cleanup rewrites any comment carrying the preview marker, with no author check #9521.
• The "file a needs-triage issue" block appears 4 times; fabrika report file already does it.
• The "only exit 0 passes" gate appears 4 times.
• The alchemy physical-name prefix is hand-built in 3 scripts.
• run-evidence.yml re-parses crabbox output in shell while src/ci/crabbox.ts owns that contract.

Caveats: publish.yml's tag grammar is read out of the yaml by publish-isolation-guard (ADR 0201), so moving it without re-pointing the guard would silently zero that gate. The changelog.yml gatherer only reads PR bodies for issue links, so a link made in the GitHub UI lands as Uncategorized.

This changes the home question from the earlier ruling note: only 3 deciding scripts are phoenix-only, so a repo-only CLI package would be very small. The package home is back with the founder.

[usirin]

Founder ruling, 2026-09-20 (later the same day): no new package

This replaces the earlier ruling note on this issue that named packages/ci-cli.

After the audit above showed only 3 of 51 deciding scripts are phoenix-only, umut ruled that the deploy failure hook from #9508 is built as a fabrika verb in packages/fabrika-cli. No packages/ci-cli, no packages/phoenix-ci, no other new package.

What that means for this issue:

• The 33 generic pipeline scripts move into packages/fabrika-cli as verbs, reusing the tested cores that already exist (src/guard/changed-files.ts, src/capture/resolve.ts, src/ci/crabbox.ts, report file).
• The 8 deploy and preview scripts are candidates for packages/anka-ops. That home is the audit's suggestion and is not ruled yet.
• The 3 phoenix-only scripts have no ruled home yet.
• Everything moved into fabrika-cli must carry no phoenix detail; values come in as arguments or config.

[usirin]

[usirin]

Triage note, 2026-09-20

Typed type:epic, p1, lane axis:pipeline-hardening, class:code. The ready-for: stamp is deliberately absent — on an epic that flip belongs to check-epic-plan, after plan-epic writes the ledger.

Why epic and not feature: the deliverable is 51 deciding scripts moving across at least three homes, two of which are still unruled. Any v1 scope I could carve to make this fit one PR would be the epic's first child, not the whole of it.

Why not type:decision: the founder already ruled the package question twice on this issue, and the later ruling (no new package, generic scripts become packages/fabrika-cli verbs) covers 33 of the 51. That is buildable now. The two open home questions are narrower than the epic and belong to it as children, not as its type.

Verified at origin/main before enriching: the scrape and the six-retry prod health loop are still inline in .github/workflows/deploy.yml after PR #9510 merged, so the gap this issue reports is live, not superseded.

Searched the board for an owner of this surface (report dedup, 1428 indexed issues): nothing owns workflow-yaml inline logic. Closest live neighbours, neither absorbing this:

• pr-cleanup rewrites any comment carrying the preview marker, with no author check #9521 — one of the three drifted sticky preview-comment writers. A live bug with its own narrow fix; left standalone rather than folded in, so it is not parked behind a three-cycle epic. plan-epic should treat it as already-covering that one copy.
• A red post-merge run on main reaches nobody: 8 failed web deploys sat unseen for 3 days #9508 — the deploy failure alert, being built as a fabrika-cli verb under the same ruling. Sibling, not a blocker; no dependency edge wired.

For the planner, the two unruled homes. The 8 deploy/preview scripts (packages/anka-ops is the audit's suggestion only) and the 3 phoenix-only scripts have no ruled home. Each should come out of plan-epic as a type:decision child left for the founder, not as build work. The 33 generic ones are ruled and can be laid out as ordinary children.

[usirin]

Folded from #9521 (backlog sweep 2026-09-26)

#9521 is closed into this issue. It is the drifted third copy of the sticky preview-comment writer that the audit here already names.

• .github/workflows/pr-cleanup.yml, step Update preview comment (torn down, ${{ matrix.app }}), picks its comment with comments.find((c) => c.body?.includes(marker)), with no author check. Both writers in deploy.yml gate on isTrustedCi (login github-actions[bot] and type Bot), added on purpose in e2e sticky-comment lookups have no author check — a forged comment can steer or skip the gate on a public repo #3740. So on PR close, any human comment that contains <!-- preview-deploy --> can be rewritten to "torn down". A PR comment that merely quotes the preview-deploy anchor shadows the real announcement #8844 shows such comments really happen.

Extra acceptance criteria for the child that builds the one tested writer:

• The writer only selects a comment authored by the Actions bot (login and type) as well as matching the marker, with a test showing that a non-bot comment carrying the marker is never picked.
• Until that writer replaces the inline copy, the cleanup step is the known unsafe one. If the epic's first children won't reach it soon, patch the one-line isTrustedCi fix into pr-cleanup.yml first.