Skip to content

test(stack): name the failing psql call when Commands stderr is empty - #7457

Merged
avallete merged 1 commit into
developfrom
avallete/flaky-fixes-2026-11-03
Nov 3, 2026
Merged

avallete merged 1 commit into
developfrom
avallete/flaky-fixes-2026-11-03

Conversation

@avallete

@avallete avallete commented Nov 3, 2026

Copy link
Copy Markdown
Member

TL;DR

Diagnostic only. The cause is still unknown, so the next failure will say which psql call lost its stderr and how that call exited.

  • Commands.integration.test.ts › podman runs SQL from stdin, streams a dump, and returns client failure: on Podman, psql exited non-zero but captured stderr was empty (expected '' to contain 'missing_column'). The vitest frame didn't show which of the two expected-failure calls fired, and stderr was shared across calls, so the report couldn't tell an attach that dropped output apart from a different exit path. Stdout and stderr are now reset before each expected psql failure, and both assertions carry the call label, the exit code and the captured stdout/stderr.

ref

spotted in: https://github.com/supabase/cli/actions/runs/41469215904/job/125358056261 (green rerun: https://github.com/supabase/cli/actions/runs/41469215904/job/125360621750)


Generated by Claude Code

Reset stdout and stderr before each expected psql failure and attach the
call label, exit code and captured output to the assertions, so a Podman
run that loses stderr shows which call it was and how psql exited.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MBEk1PhFmm5bGsyrcpQfsk
@avallete
avallete requested a review from a team as a code owner November 3, 2026 07:40

avallete commented Nov 3, 2026

Copy link
Copy Markdown
Member Author

/ai-review


Generated by Claude Code

@avallete avallete added the run-flaky-check Run the flaky-test check on this PR label Nov 3, 2026 — with Claude
@github-actions github-actions Bot removed the run-flaky-check Run the flaky-test check on this PR label Nov 3, 2026
@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

CLI benchmark (Lightweight Commands): no changes found

Commit 443e734 against develop at c9473a7 · 10 runs each · report only, never blocks merging

Command Workload Before After Change
supabase --version version 150 ms 151 ms no real change
supabase --help help 150 ms 150 ms no real change
supabase init init 152 ms 153 ms no real change
supabase migration new bench migration-new 159 ms 161 ms no real change
supabase gen types --lang typescript --project-id replayfixtureprojref gen-types 179 ms 178 ms no real change
supabase projects list projects-list 163 ms 167 ms no real change
supabase functions list --project-ref replayfixtureprojref functions-list 172 ms 170 ms no real change

API calls: unchanged for all 7 commands.

Binary size: 134.0 MB, unchanged.

Details: timing ranges, API calls per command, slowest phases

Timing ranges

Command Workload Before range After range Change Verdict
supabase --version version 147-162 ms 148-162 ms +0 ms (+0%) within noise
supabase --help help 148-155 ms 147-155 ms 0 ms (0%) within noise
supabase init init 150-156 ms 148-160 ms +1 ms (+1%) within noise
supabase migration new bench migration-new 156-168 ms 156-176 ms +1 ms (+1%) within noise
supabase gen types --lang typescript --project-id replayfixtureprojref gen-types 174-202 ms 176-184 ms -1 ms (-1%) within noise
supabase projects list projects-list 160-177 ms 161-170 ms +4 ms (+2%) within noise
supabase functions list --project-ref replayfixtureprojref functions-list 164-176 ms 166-177 ms -2 ms (-1%) within noise

API calls per command

Command Workload Before After Endpoint changes
supabase --version version 0 0
supabase --help help 0 0
supabase init init 0 0
supabase migration new bench migration-new 0 0
supabase gen types --lang typescript --project-id replayfixtureprojref gen-types 2 2
supabase projects list projects-list 1 1
supabase functions list --project-ref replayfixtureprojref functions-list 2 2

Slowest phases

No command moved beyond noise, so there are no phase deltas.

Workflow run

Comment /bench to run the lightweight benchmark (this comment) again on the latest commit, or /bench full to also run the integration-heavy benchmark, which posts in its own comment.

@github-actions github-actions Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superseded by a newer AI review

🤖 AI Review

Both independent reviews reported no findings. Code inspection confirms that the buffer resets isolate each psql failure and the diagnostic messages preserve existing assertions. No additional issues found. Tests were not run.

Findings

No issues found.

Stats

Claude findings: 0 · Codex findings: 0 · Confirmed: 0 · Refuted: 0 · Uncertain: 0


Models: claude-opus-5-5 + gpt-6.1-sol · Trigger: auto · Workflow run

This review runs once per PR. A maintainer can request another with a /ai-review comment.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI Review

Both independent reviews reported no findings. Reading the full diff and surrounding checked-out code confirmed that the changes isolate captured output and add failure diagnostics without weakening assertions. No issues found. Tests were not run.

Findings

No issues found.

Stats

Claude findings: 0 · Codex findings: 0 · Confirmed: 0 · Refuted: 0 · Uncertain: 0


Models: claude-opus-5-5 + gpt-6.1-sol · Trigger: manual · Workflow run

This review runs once per PR. A maintainer can request another with a /ai-review comment.

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Flaky test check: ✅ no flaky tests

Commit 5416379cef4e · focused ×10 (5 executions each), 6 tests · integration ×10, 6695 tests · unit ×10, 9353 tests · workflow run

Each execution is a separate Vitest process; Where lists failing runs, as run.iteration for suites executed more than once per run. The full data is in the flaky-check-report artifact.

@avallete
avallete added this pull request to the merge queue Nov 3, 2026
Merged via the queue into develop with commit ba71d22 Nov 3, 2026
93 checks passed
@avallete
avallete deleted the avallete/flaky-fixes-2026-11-03 branch November 3, 2026 09:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants