Skip to content

fix(stack): include recent service output in startup failure errors - #6791

Merged
avallete merged 8 commits into
developfrom
avallete/startup-failure-output-08dd39
Sep 24, 2026
Merged

avallete merged 8 commits into
developfrom
avallete/startup-failure-output-08dd39

Conversation

@avallete

@avallete avallete commented Sep 24, 2026 •

Copy link
Copy Markdown
Member

Summary

When a stack service's startup step fails (for example Realtime's migration step), the error now shows the last 20 lines the process printed on each of stdout and stderr. Before, it showed only stderr, which was usually the final exception without its cause. A startup step that hangs now also reports its recent output, including a last line that doesn't end in a newline, instead of a bare timeout.

Why

In #6775, supabase start under SUPABASE_EXPERIMENTAL_STACK=1 failed with only Realtime's final Ecto pool-timeout exception:

realtime startup exited with 1: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue ...

The actual cause went to stdout on every connection attempt and was never shown:

[error] Postgrex.Protocol ("db_conn_1") failed to connect: ** (DBConnection.ConnectionError) tcp connect (host.docker.internal:54322): network is unreachable - :enetunreach

supabase stack logs couldn't recover it either, because nothing is retained after a failed start.

Before

flowchart LR
  P[startup process<br/>e.g. realtime prepare] -->|stdout<br/>Postgrex :enetunreach ×N| O[buffered in full,<br/>published after exit]
  P -->|stderr<br/>final Ecto exception| E[buffered in full]
  E --> M["❌ realtime startup exited with 1:<br/>&lt;stderr only&gt;"]
  O -. dropped from the error .-> X((root cause lost))
  P -->|hangs &gt; 60s| T["❌ Service launch timed out<br/>(no output at all)"]
Loading

After

flowchart LR
  P[startup process<br/>e.g. realtime prepare] -->|stdout chunks| L[published to service logs<br/>as they arrive]
  P -->|stderr chunks| L
  P -->|stdout lines| RO[stdout tail<br/>last 20 non-blank lines]
  P -->|stderr lines| RE[stderr tail<br/>last 20 non-blank lines]
  RO --> R{startup result}
  RE --> R
  R -->|exit ≠ 0| M["❌ realtime startup exited with 1<br/>Recent stdout:<br/>[error] Postgrex ... :enetunreach<br/>Recent stderr:<br/>** (DBConnection.ConnectionError) ..."]
  R -->|hangs &gt; 60s| T["❌ realtime startup timed out after 60 seconds<br/>Recent stdout: ...<br/>Recent stderr: ..."]
Loading

What changed

  • awaitStartup in packages/stack/src/services/ProcessRecipe.ts splits stdout and stderr into lines and keeps a separate tail of the last 20 non-blank lines for each, so later stdout noise can't push out the stderr error.
  • Each retained line is clipped to its last 1000 characters, so memory stays bounded even for output without newlines.
  • Each stream's unfinished last line is added to its tail when the stream ends or is interrupted, so a hung process's last message still appears after the 60-second timeout.
  • Native and container startup paths share one failure formatter. A non-zero exit and a timeout both append Recent stdout: / Recent stderr: sections for whichever streams printed something.
  • Startup output is published to the service's log stream as it arrives, the same way the main service process already works, instead of in one piece after exit.

Linked issue

Part of #6775

🤖 Generated with Claude Code

Startup failures reported only stderr, hiding root causes that services
print to stdout (e.g. Realtime's Postgrex connection errors). Keep a
bounded tail of the last 20 lines of combined stdout/stderr, including an
unterminated last line, and append it to both non-zero exit and timeout
errors. Startup output is now published to the service logs as it arrives.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@avallete
avallete requested a review from a team as a code owner September 24, 2026 10:32

@github-actions github-actions Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superseded by a newer AI review

🤖 AI Review

The startup output change improves diagnostics, but the retained output is not fully bounded, and later stdout can evict a useful stderr error. The timeout duration is also duplicated. All four reported findings were verified and merged into three distinct findings.

Findings

Severity Location Category Sources Claim
🟡 MINOR packages/stack/src/services/ProcessRecipe.ts:238 resource-usage claude+codex A newline-free startup output stream can grow the partial-line buffer without limit and produce an arbitrarily large error message.
🟡 MINOR packages/stack/src/services/ProcessRecipe.ts:220 error-handling codex Later stdout lines can evict the stderr message explaining a startup failure.
⚪ NIT packages/stack/src/services/ProcessRecipe.ts:251 maintainability claude The startup timeout duration is duplicated in the timer and error text, so future edits could make them disagree.

Stats

Claude findings: 2 · Codex findings: 2 · Confirmed: 3 · Refuted: 0 · Uncertain: 0


Models: claude-opus-5-5 + gpt-6-sol · Trigger: auto · Workflow run

This review runs once per PR. A maintainer can request another with a /ai-review comment.

Comment thread packages/stack/src/services/ProcessRecipe.ts
Comment thread packages/stack/src/services/ProcessRecipe.ts Outdated
Comment thread packages/stack/src/services/ProcessRecipe.ts Outdated
avallete and others added 2 commits September 24, 2026 12:44
Later stdout could evict the stderr error from a shared tail, and a
newline-free stream could grow the retained line without limit. Keep a
20-line tail per stream, clip each retained line to its last 1000
characters, and derive the timeout message from the timer's constant.
The test fake also implements the discard/kill process members from
develop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@avallete

Copy link
Copy Markdown
Member Author

/ai-review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI Review

The startup output change has two confirmed low-impact issues: collected output is omitted when a stream or exit-code effect fails, and clipping can split a Unicode surrogate pair. Codex's independent review reported no findings.

Findings

Severity Location Category Sources Claim
🟡 MINOR packages/stack/src/services/ProcessRecipe.ts:265 error-handling claude If a startup output stream or the exit-code effect fails, the error omits output collected before the failure.
⚪ NIT packages/stack/src/services/ProcessRecipe.ts:213 correctness claude Clipping a startup output line can split a UTF-16 surrogate pair, leaving an unpaired surrogate in the error message.

Stats

Claude findings: 2 · Codex findings: 0 · Confirmed: 2 · Refuted: 0 · Uncertain: 0


Models: claude-opus-5-5 + gpt-6-sol · Trigger: manual · Workflow run

This review runs once per PR. A maintainer can request another with a /ai-review comment.

Comment thread packages/stack/src/services/ProcessRecipe.ts
Comment thread packages/stack/src/services/ProcessRecipe.ts Outdated

@jgoux jgoux left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new startup test fixture fails the package type check because it is missing a required ContainerRuntime member.

Comment thread packages/stack/src/services/ProcessRecipe.integration.test.ts
@jgoux
jgoux self-requested a review September 24, 2026 13:36
@avallete
avallete enabled auto-merge September 24, 2026 13:36
@avallete
avallete added this pull request to the merge queue Sep 24, 2026
Merged via the queue into develop with commit 53db9b5 Sep 24, 2026
21 checks passed
@avallete
avallete deleted the avallete/startup-failure-output-08dd39 branch September 24, 2026 14:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants