[Prashansa-K]

Adds a non-blocking job that compares the CLI build of each ready pull request into develop with a build of the develop commit it branches from, without Docker, a database, or the network. It writes a job summary that leads with the answer ("no changes found" or "N things to check"), then one plain-words line per flag (including when a command that made no Management API calls starts making them), a before/after timing table per command, API call counts, binary size, and a collapsed section with timing ranges, per-command API calls, and the slowest phases for commands that moved.

The first push to a pull request from this repository benchmarks and posts one comment with the same report, tagged with a hidden marker. Later pushes skip the benchmark while that comment reports a result, and the job summary says so. Pull requests from forks have a read-only token, so they get the job summary only and benchmark every push.

A maintainer can comment /bench to run the benchmark again on the pull request's current head commit. The command re-runs that commit's workflow run (the same as "Re-run all jobs" in the Actions UI), and the re-run updates the comment in place. An eyes reaction on the /bench comment means GitHub accepted the re-run; the result arrives as the updated benchmark comment, whether it succeeded or failed. If nothing can be re-run, the pull request gets a reply saying why: a benchmark is already running, the run is older than 30 days, GitHub refused the re-run, or there is no run for the commit. The comment keeps a collapsed "Previous runs" list (commit, time, headline, link; up to 20) in a hidden marker. Only the repository owner and people with write access can run it, and only on open, non-draft pull requests into develop from this repository. The whole comment must be exactly /bench; any other comment is ignored. Comment-triggered workflows run from the default branch, so /bench starts working only after this merges to develop.

A pull request that changes none of the benchmarked paths, or more than the 300 files GitHub's path filter inspects, gets no run, so /bench replies that there is none.

• apps/cli/scripts/bench-cli.ts gains --workloads <manifest> and --api-url. Every launch gets a profile file pointing at a local replay server (the profile is the CLI's only API redirect), and the harness resets the server's request log and scenario before each launch. API calls are counted from HTTP client spans in the trace and cross-checked against the server's request log.
• apps/cli/scripts/bench-report.ts holds the report schemas and pure analysis; bench-summary.ts renders the Markdown for the job summary and the comment; replay-control.ts is the client for the replay server's control API.
• apps/cli/bench/workloads.pr.json lists the workloads. They were chosen from command usage data, ranked and filtered to commands that need no Docker, database, network, or prompt; the method, query, and caveats are in apps/cli/docs/tracing-monitoring.md.
• apps/cli-e2e/scripts/replay-serve.ts starts the existing replay server standalone.
• .github/workflows/cli-bench.yml and .github/scripts/cli-bench/ are new. No existing workflow or action is edited.

The /bench job never checks out or runs pull request code: it runs default branch code and only asks GitHub to re-run the existing pull_request run, so benchmarks always execute in the pull request's own run and cache scope, like any other pull request job. Permissions are split by job: the job that decides whether to benchmark is read-only, the /bench job is the only one with actions: write, the job that builds and runs pull request code has read-only permissions, and a separate job, the only pull_request job with comment-write permission, downloads the rendered markdown, sanitizes it, and posts it; it never runs pull request code. Whether the workflow token may call the re-run endpoint is verified on the first /bench after merge; a refusal is reported in the reply.

Pull request runs now measure the pull request head commit instead of the test merge commit, so the commit shown in the report is the one that was benchmarked.

🤖 Generated with Claude Code

[github-actions]

🤖 AI Review

Both independent reviews were available. Code inspection confirms eight distinct findings: four minor reporting/failure-handling issues and four nits. No critical or major findings; no duplicates.

Findings

Severity	Location	Category	Sources	Claim
🟡 MINOR	apps/cli/scripts/bench-cli.ts:373	error-handling	codex	A failed traced launch loses its exit code when no usable spans are decoded, allowing a timing verdict despite that failed invocation.
🟡 MINOR	apps/cli/scripts/bench-report.ts:396	correctness	codex	Unavailable API measurements become zero endpoint counts, producing false endpoint additions or removals.
🟡 MINOR	.github/workflows/cli-bench.yml:54	ci	codex	A branch-binary build failure fails the benchmark job and suppresses its explanatory summary, contrary to the documented report-only behavior.
🟡 MINOR	.github/workflows/cli-bench.yml:102	error-handling	claude	The benchmark runs after replay-server startup fails and then reports a misleading secondary failure.
⚪ NIT	.github/workflows/cli-bench.yml:75	ci	claude	The merge-base dependency install omits HUSKY=0, allowing hook installation to modify Git configuration shared with the main checkout.
⚪ NIT	apps/cli/scripts/bench-summary.ts:181	error-handling	claude	Report read and schema-decoding errors are silently presented as a missing report.
⚪ NIT	apps/cli/scripts/bench-summary.ts:111	correctness	claude	The phase-summary sentence incorrectly says no workload had a timing verdict whenever any workload is not comparable and no phase rows exist.
⚪ NIT	apps/cli/scripts/bench-cli.ts:204	error-handling	claude	Workload validation fails with a plain string, unlike the tagged error used by the new replay-control helper.

Stats

Claude findings: 5 · Codex findings: 3 · Confirmed: 8 · Refuted: 0 · Uncertain: 0

---

Models: claude-opus-5-5 + gpt-6.1-sol · Trigger: auto · Workflow run

This review runs once per PR. A maintainer can request another with a /ai-review comment.

[github-actions]

CLI benchmark: no changes found

Commit 0a1dff3 against develop at 5e7ad29 · 10 runs each · report only, never blocks merging

Command	Before	After	Change
supabase --version	225 ms	221 ms	no real change
supabase --help	180 ms	173 ms	no real change
supabase init	191 ms	179 ms	no real change (-6%, within normal variation)
supabase migration new bench	202 ms	199 ms	no real change
supabase gen types --lang typescript --project-id replayfixtureprojref	263 ms	249 ms	no real change (-5%, within normal variation)
supabase projects list	343 ms	337 ms	no real change
supabase functions list --project-ref replayfixtureprojref	328 ms	318 ms	no real change

API calls: unchanged for all 7 commands.

Binary size: 133.8 MB, unchanged.

Details: timing ranges, API calls per command, slowest phases

Timing ranges

Command	Before range	After range	Change	Verdict
supabase --version	196-334 ms	175-249 ms	-4 ms (-2%)	within noise
supabase --help	166-194 ms	159-194 ms	-7 ms (-4%)	within noise
supabase init	171-224 ms	166-206 ms	-11 ms (-6%)	within noise
supabase migration new bench	179-256 ms	185-240 ms	-3 ms (-1%)	within noise
supabase gen types --lang typescript --project-id replayfixtureprojref	204-408 ms	193-429 ms	-14 ms (-5%)	within noise
supabase projects list	279-397 ms	312-374 ms	-5 ms (-2%)	within noise
supabase functions list --project-ref replayfixtureprojref	277-435 ms	290-434 ms	-10 ms (-3%)	within noise

API calls per command

Command	Before	After	Endpoint changes
supabase --version	0	0
supabase --help	0	0
supabase init	0	0
supabase migration new bench	0	0
supabase gen types --lang typescript --project-id replayfixtureprojref	2	2
supabase projects list	1	1
supabase functions list --project-ref replayfixtureprojref	2	2

Slowest phases

No command moved beyond noise, so there are no phase deltas.

Comment /bench to run it again on the latest commit.

Workflow run

[Coly010]

requesting changes for one thing, the rest is small

1. i think /bench might open up develop's cache to PR code. you've already called out the runner's cache token in the description, and i agree that for same-repo authors on normal pull_request runs it's a wash. where i'm less sure is the issue_comment path. that run is in develop's scope, and the bench job checks out the PR head and runs its own copy of .github/actions/setup, so dependency-cache: false only holds if the PR's action.yml honours it (and the build steps are in develop's cache scope either way). the release jobs in release.yml / release-shared.yml restore develop-scope caches on ubuntu-latest, and the pnpm store one falls back to a pnpm-store-files-<os>-<arch>- prefix match. so a same-repo branch (a renovate one included, if someone types /bench on it) could plant a store that the next release restores without it ever being reviewed. i haven't tried it end to end, but pushing a branch normally can't get anything into a release job

   would it work to keep /bench but have it re-run the PR's own benchmark run, instead of benchmarking inside the comment run? something like:

   • resolve stays as it is on issue_comment (develop's code, same permission checks), finds the latest pull_request run of this workflow for the PR's current head, and re-runs it through the API (needs actions: write on resolve only)
   • a re-run keeps the original run's context, so the cache stays scoped to the PR and a fork's token stays read-only
   • the comment job can upsert whenever github.run_attempt > 1, which covers /bench and the re-run button in the Actions tab
   • if the latest push didn't touch the paths filter there's no run for the current head, so /bench could just reply saying so

   that way the issue_comment path only ever runs develop's code, and PR code never gets develop's cache scope. i'm fairly sure a GITHUB_TOKEN can trigger a re-run, but it's worth confirming with one run

2. smaller one: runLaunch in bench-cli.ts has no timeout, so one hung launch sits until the 30 min job timeout and the summary loses every workload that already finished. an Effect.timeout per launch that records a non-zero exit would land in the existing not-comparable path

happy to be told i'm missing something on 1. otherwise, it looks good to me!

[Prashansa-K]

Fixed the PR based on the two suggestions. Added in-depth description in the PR description.
TL;DR:

1. Using the re-run approach as suggested in the review comment for /bench.
2. Timeout of 60s added for cli-launch.