Skip to content

test(stack): fix 1 flaky test in the Realtime catalog check - #7359

Open
avallete wants to merge 2 commits into
developfrom
avallete/flaky-fixes-2026-10-28
Open

avallete wants to merge 2 commits into
developfrom
avallete/flaky-fixes-2026-10-28

Conversation

@avallete

Copy link
Copy Markdown
Member

TL;DR

The Realtime, pg-meta and Studio catalog test hung on Podman without saying where. It now fails before the test timeout, naming the service and phase that stalled and that service's output.

  • Realtime.integration.test.ts › serves Realtime, pg-meta, and Studio against the owned database: on Podman, the test hit the 120s test timeout with only the generator start frame as a clue. The test runs four services in sequence (database, Realtime, pg-meta, Studio), each with a recipe, start, readiness and HTTP step, so a stall anywhere looked the same. The cause is still unknown, so this change is a diagnostic. The scoped body now runs under a 110s guard. Each step records its phase, and each recipe's launch output is collected. A stall fails with CatalogStallError naming the service and phase in progress (including stop and root cleanup) and the last part of that service's output.

ref

spotted in:


Generated by Claude Code

The Realtime, pg-meta and Studio catalog test hit the 120s test timeout
on Podman with only the generator frame to go on. It now runs under a
110s guard that fails with CatalogStallError naming the service and
phase in progress, plus that service's launch output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MBEk1PCdaXDCUPR16M9Ria
@avallete
avallete requested a review from a team as a code owner October 28, 2026 07:41

Copy link
Copy Markdown
Member Author

/ai-review


Generated by Claude Code

@avallete avallete added the run-flaky-check Run the flaky-test check on this PR label Oct 28, 2026 — with Claude
@github-actions github-actions Bot removed the run-flaky-check Run the flaky-test check on this PR label Oct 28, 2026

@github-actions github-actions Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superseded by a newer AI review

🤖 AI Review

Verified all four reported findings and merged the duplicate timeout reports. Confirmed one minor cleanup-diagnostic issue and two diagnostic nits. Updated stale line references to match the checked-out code.

Findings

Severity Location Category Sources Claim
🟡 MINOR packages/stack/src/services/Realtime.integration.test.ts:211 test-reliability claude+codex The timeout guard waits for scoped finalization, so a hanging teardown can prevent CatalogStallError from reaching the test runner before its timeout.
⚪ NIT packages/stack/src/services/Realtime.integration.test.ts:209 diagnostics claude Some phase labels identify a narrower operation than the work actually running, which can misdirect stall diagnosis.
⚪ NIT packages/stack/src/services/Realtime.integration.test.ts:43 correctness claude Decoding each output chunk independently can corrupt split UTF-8 characters, and trimming by UTF-16 code units can leave a dangling surrogate.

Stats

Claude findings: 3 · Codex findings: 1 · Confirmed: 3 · Refuted: 0 · Uncertain: 0


Models: claude-opus-5-5 + gpt-6.1-sol · Trigger: manual · Workflow run

This review runs once per PR. A maintainer can request another with a /ai-review comment.

Comment thread packages/stack/src/services/Realtime.integration.test.ts
Comment thread packages/stack/src/services/Realtime.integration.test.ts Outdated
Comment thread packages/stack/src/services/Realtime.integration.test.ts Outdated

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI Review

Merged four findings into three. Confirmed the phase-label and UTF-8 decoding nits. The cleanup-timeout concern remains uncertain: scope placement and unbounded cleanup are verified, but the pinned Effect implementation is unavailable, and the patched Vitest runner complicates the claimed diagnostic loss.

Findings

Severity Location Category Sources Claim
🟡 MINOR packages/stack/src/services/Realtime.integration.test.ts:211 test-diagnostics claude+codex Stalled scope cleanup may prevent the 110-second guard from surfacing CatalogStallError before Vitest's 120-second timeout.
⚪ NIT packages/stack/src/services/Realtime.integration.test.ts:35 test-diagnostics claude Filesystem root setup is labeled 'database recipe', so a stall during that setup would identify the wrong phase.
⚪ NIT packages/stack/src/services/Realtime.integration.test.ts:43 correctness codex Decoding each output chunk with a fresh TextDecoder corrupts UTF-8 characters split across chunk boundaries.

Stats

Claude findings: 2 · Codex findings: 2 · Confirmed: 2 · Refuted: 0 · Uncertain: 1


Models: claude-opus-5-5 + gpt-6.1-sol · Trigger: auto · Workflow run

This review runs once per PR. A maintainer can request another with a /ai-review comment.

Comment thread packages/stack/src/services/Realtime.integration.test.ts
Comment thread packages/stack/src/services/Realtime.integration.test.ts Outdated
Comment thread packages/stack/src/services/Realtime.integration.test.ts Outdated
The guard now joins a detached body under its deadline, so a teardown
that hangs during interruption cannot delay the stall report. Phase
labels cover root setup, the pg-meta relay and scope teardown, and
launch output is decoded as a stream per channel.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MBEk1PCdaXDCUPR16M9Ria
@avallete avallete added the run-flaky-check Run the flaky-test check on this PR label Oct 28, 2026 — with Claude
@github-actions

github-actions Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Flaky test check: ✅ no flaky tests

Commit e9d829dcbef2 · focused ×10 (5 executions each), 1 test · integration ×10, 5635 tests · unit ×10, 8394 tests · workflow run

Each execution is a separate Vitest process; Where lists failing runs, as run.iteration for suites executed more than once per run. The full data is in the flaky-check-report artifact.

@github-actions github-actions Bot removed the run-flaky-check Run the flaky-test check on this PR label Oct 28, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants