Skip to content

fix: Web deploy reds on every main push: the prod health gate finds no worker URL - #9510

Merged
usirin merged 1 commit into
mainfrom
build/9507-deploy-url-scrape-7b9f1d9f
Sep 20, 2026
Merged

usirin merged 1 commit into
mainfrom
build/9507-deploy-url-scrape-7b9f1d9f

Conversation

@usirin

@usirin usirin commented Sep 20, 2026

Copy link
Copy Markdown
Member

The prod deploy job has reddened on every push to main since the Alchemy upgrade (#8934). The
deploy itself succeeds; the health gate after it fails because the URL scrape returns nothing.

deploy.yml scraped the deployed URL by matching a *.workers.dev host anywhere in deploy.log.
Alchemy prints a summary object whose url: key carries the custom domain when the stage has one,
and it no longer prints a workers.dev host beside it for prod. So the host-shaped pattern matched
nothing, steps.deploy.outputs.url came back empty, and the verify step hit its fail-closed
empty-URL branch before its first probe.

This matches on the url: key instead of on the host shape, which covers both a custom domain and
a workers.dev host with one pattern:

echo "url=$(sed -nE "s|^[[:space:]]*url: '(https://[^']+)'.*|\1|p" deploy.log | tail -1)" >> "$GITHUB_OUTPUT"

The empty-URL branch and the step's if: github.event_name == 'push' guard are untouched — no
outputs.url != '' clause was added anywhere (#1617, ADR 0092). An unscrapeable URL still reds the
job.

The widened pattern against real Alchemy output

Red prod run 35531984366, job 106134117846 (the run the issue names) printed:

{
  url: 'https://phoenix.kamp.us',
  domains: [ 'phoenix.kamp.us' ],
  databaseId: 'a2c9230c-1115-4f5d-9867-3e3ece7a1e82',
  accountId: '***'
}

Run against that block, the new pattern extracts https://phoenix.kamp.us. The old pattern extracts
the empty string from it — that is the bug. Probing the custom domain is also the stronger check:
it is the host users actually hit.

I also ran the pattern against a log with no url: line at all. It yields the empty string, so the
fail-closed branch still fires on a genuinely URL-less deploy.

The preview regex in ci.yml is settled: no widening

Preview deploy log: https://github.com/kamp-us/phoenix/actions/runs/35476125242/job/105985669521
(PR #9279's preview, pr-9279 stage). Its Alchemy output:

{
  url: 'https://phoenix-phoenix-pr-9279-sfzd4ajx67y4nywa.kampusinfra.workers.dev',
  domains: [],
  databaseId: '0a55e11f-79d2-4f80-8096-69950bfa2f5b',
  accountId: '***'
}

A pr-<n> stage binds no custom domain — domains: [] — so Alchemy still prints a workers.dev host
there, and the same job's PREVIEW_URL step env carried that host. The webRe in ci.yml is
therefore not broken and is left unchanged. Widening it would be worse than a no-op: it would let a
custom-domain line, which only prod produces, satisfy a preview poll. I recorded that reasoning as a
comment beside the regex so the next reader does not re-open it.

Note that the preview comment's URL comes from this same steps.deploy.outputs.url, so the scrape
change does reach previews — and the new pattern extracts the identical workers.dev host from the
preview block above, so the comment's contents do not move.

Deviations

  • Out-of-scope change — Said: the issue's fix is in .github/workflows/deploy.yml, and
    ci.yml is to be changed only if a preview URL can be non-workers.dev. Did: left the ci.yml
    regex unchanged but added a four-line comment above it recording why the workers.dev shape is
    deliberate. Why: an acceptance criterion asks for a comment or a PR-body line recording the
    finding, and a PR body is not a surface the next reader of that regex will find.
    Disposition: stated here; comment only, no behavior change.

Fixes #9507

@github-actions

github-actions Bot commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

🚀 Preview deployed

  • web — Stage pr-9510 torn down.

@usirin

usirin commented Sep 20, 2026

Copy link
Copy Markdown
Member Author

governance: PASS @ b34bbc4 content:838f8e480422 — no contradiction, no weakening

Derivation

governance scope derives the namespace as required at b34bbc4 over governedRoots as declared in .fabrika.jsonc: both changed files sit under the .github/ root. self false — this diff edits neither this skill nor its contract, so the self fence does not apply and the head's rules were used.

Whether this diff also needs a code-owner approval is a separate question CODEOWNERS answers, not this one.

Corpus half — does this contradict standing law

No decision record is in the diff, so there was no subject to rank and sweep was not run. The corpus half is a hand read of the standing records in this domain.

Questions this change answers:

  1. May the prod health gate's URL input be scraped by a key-anchored pattern instead of a host-shaped one? — Yes. Nothing in the corpus binds the gate to a *.workers.dev host; what is bound is the gate's fail-closed posture on an empty URL.
  2. Does an unscrapeable URL still red the prod deploy? — Yes, unchanged.
  3. Must the preview poll's URL shape widen alongside it? — No, and the PR settles it on a read preview log rather than on assumption.

Records read by hand:

No contradiction found.

Gate half — does this quietly weaken a guard

governance guards at b34bbc4: 0 anchored invariants in reach, 2 guard-bearing files compared block-by-block against the base — .github/workflows/ci.yml and .github/workflows/deploy.yml. Both guard files carry unanchored invariants, which is the gap a hand read covers; here is that read.

deploy.yml — the URL scrape. The pattern moves from grep -oE 'https://[A-Za-z0-9.-]+\.workers\.dev' (unanchored, anywhere in the log) to sed -nE "s|^[[:space:]]*url: '(https://[^']+)'.*|\1|p" (anchored to line start on Alchemy's url: key). Broadening the accepted host is not a softening here, because the new pattern is strictly narrower in where it may match: the old one would take a workers.dev host from any line of deploy.log, the new one only from a line whose leading token is the url: key. The probe target's trust boundary is unchanged either way — it is the trusted deploy job's own stdout on a push to main.

deploy.yml — the fail-closed branch. Untouched, and verified at head rather than inferred from the diff's silence: if [ -z "$PROD_URL" ] still exits 1 with the #1617 error, the retry exhaustion still exits 1 with the #1436 error, and the step if: is still the bare event guard. I also ran the new pattern against a URL-less log fixture — it yields the empty string, so the fail-closed branch still fires on a genuinely URL-less deploy. The empty case survives the change.

ci.yml — the preview poll. Comment-only. The webRe regex, the isTrustedCi author check and the CI_COMMENT_AUTHOR binding (#3740) are byte-identical. The added comment documents a constraint against future widening — that a custom-domain line, which only prod produces, must not be allowed to satisfy a preview poll — so its effect on the guard is to harden the rationale, never to soften the check.

No weakening found; no invariant in this diff's reach was removed or softened.

Verdict

PASS — no contradiction with standing law, no guard weakened. Everything the two halves needed was read.

@usirin

usirin commented Sep 20, 2026

Copy link
Copy Markdown
Member Author

review-code: PASS @ b34bbc4 content:838f8e480422 — merge-ready

Scoped at b34bbc4 — 2 changed files, class code only, fixes:9507, governance: required, self false. No prior verdicts at this head.

Acceptance criteria

1. The URL scrape extracts the deployed URL from Alchemy's url: line, matching a custom domain as well as a *.workers.dev host — MET.

deploy.yml now scrapes with sed -nE "s|^[[:space:]]*url: '(https://[^']+)'.*|\1|p" deploy.log | tail -1. I ran that exact pattern against both log shapes:

  • url: 'https://phoenix.kamp.us', → https://phoenix.kamp.us
  • url: 'https://phoenix-phoenix-pr-9279-sfzd4ajx67y4nywa.kampusinfra.workers.dev', → that host

The old grep -oE 'https://[A-Za-z0-9.-]+\.workers\.dev' yields the empty string on the first, which is the bug the issue reports. One pattern serves both stages, as the criterion asks.

The quoting survives the shell: \1 inside the double-quoted sed script within $( ) inside echo "…" stays a literal backreference, and tail -1 guarantees the single line $GITHUB_OUTPUT needs. The pattern is anchored to line start, so the workflow's own #-prefixed comments — which the runner echoes to the job log, not into deploy.log — cannot be matched either way.

2. steps.deploy.outputs.url is non-empty for the prod web deploy, so the verify step probes /api/health — MET as far as a pre-merge read can carry it. The reproduction above extracts a non-empty URL from the actual red run's block, so the empty-URL branch no longer fires on that input. Probing https://phoenix.kamp.us is also the stronger check: it is the host users hit.

3. The empty-URL branch still exits non-zero and the if: guard stays github.event_name == 'push' alone (#1617, ADR 0092) — MET, verified at head rather than inferred from the diff's silence. Read out of b34bbc4, the verify step's guard is the bare event guard with no outputs.url != '' clause, and the if [ -z "$PROD_URL" ] branch still prints the #1617 ::error:: and exit 1. I also ran the new pattern against a log with no url: line at all: it yields the empty string, so the fail-closed branch still fires on a genuinely URL-less deploy.

4. The ci.yml preview regex is settled either way, with a comment or PR-body line recording which — MET. [evidence: a preview deploy log linked in the PR description] — I fetched that job's log myself rather than taking the PR body's excerpt for it. Job 105985669521 (run 35476125242, deploy (web, @kampus/web, true), conclusion success) prints at line 662:

  url: 'https://phoenix-phoenix-pr-9279-sfzd4ajx67y4nywa.kampusinfra.workers.dev',
  domains: [],

and that same job's PREVIEW_URL step env at line 813 carries that host. The PR body's quoted block matches the log verbatim. So a pr-<n> stage binds no custom domain, the preview URL is still workers.dev, and leaving webRe unchanged is the correct settlement. The four-line comment beside the regex records it where the next reader will find it.

5. The widened pattern is shown to match real Alchemy output, with the red prod block and the extracted URL in the PR description — MET. [evidence: the deploy-log excerpt and extracted URL in the PR description] — again read at source, not relayed. Job 106134117846 (run 35531984366, conclusion failure, head 0cb9588f) prints at line 609:

  url: 'https://phoenix.kamp.us',
  domains: [ 'phoenix.kamp.us' ],

and at line 654 the job fails with prod health gate FAILED (issue #1617) — alchemy deploy exited 0 but emitted no scrapeable worker URL. The PR body's excerpt matches, and the URL it claims the new pattern extracts is the one I reproduced.

All five criteria are discharged. Nothing is left open.

Findings

None in scope. The fan-out on silent failure, type design and test gaps:

  • Silent failure — the change's whole subject. The fail-closed branch is preserved and independently re-tested against a URL-less fixture; the fix removes a permanently-red gate rather than softening it.
  • Type design — n/a on a workflow diff. The nearest analogue, the single-line $GITHUB_OUTPUT contract, is held by tail -1.
  • Test gaps — a workflow scrape has no unit surface in this repo; the evidence standard the issue set (reproduce against the real log block, both directions) is the right one and was met. Not worth an acceptance criterion.

Comments

The eight added lines in deploy.yml and four in ci.yml are load-bearing under this repo's comment rule: each records a constraint the code cannot express — why the match is on the url: key and not the host shape, and why the preview regex must not be widened. Both are exactly the regression the next reader would otherwise re-introduce. The replaced deploy.yml comment, which asserted the now-false url: 'https://…workers.dev' shape, is gone rather than left stale. writing-for-agents applied to both blocks: plain, specific, no restated headers.

Deviations

Disclosed Substance Verified
Out-of-scope change — ci.yml touched when the issue scoped the fix to deploy.yml Left webRe byte-identical, added a comment recording why the workers.dev shape is deliberate Matches. The ci.yml hunk is comment-only; the regex, the isTrustedCi author check and the CI_COMMENT_AUTHOR binding (#3740) are unchanged. Criterion 4 explicitly invites a comment, so this is disclosed scope rather than drift.

deviation-disclosure: PASS — nothing undisclosed that this gate could see.

CI at head

settled / green — 45 check runs, 43 success, 2 skipped, 28 of 40 repo-authored workflows inspected b34bbc4.

Governance

Fired this round; governance PASS is posted at this head as its own namespace.

Verdict

PASS.

Verdict-written: 2026-09-20T19:58:47Z

@usirin

usirin commented Sep 20, 2026

Copy link
Copy Markdown
Member Author

ship: AWAITING-CP-APPROVAL — PR #9510 @ b34bbc4 → human

ship cp-approval reads stop awaiting-approval: 4 control-plane owners resolved, 0 approving reviews at this head. This PR touches a control-plane surface, so it needs an approval from a declared owner before any gate or enqueue runs.

No base drift was reported, so no rebase is owed. Merge intent: cleared (nothing was armed). Branch untouched.

@usirin
usirin added this pull request to the merge queue Sep 20, 2026
Merged via the queue into main with commit 25ba754 Sep 20, 2026
47 checks passed
@usirin
usirin deleted the build/9507-deploy-url-scrape-7b9f1d9f branch September 20, 2026 20:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Web deploy reds on every main push: the prod health gate finds no worker URL

2 participants