[Coly010]

This gives the CLI suite its own cadence: one model daily across every channel, plus a small model matrix weekly. It's also the ask from our side, since it touches eval-refresh.yml, which is yours. If you'd rather we took an agreed slot in the shared nightly instead, that's totally fine and I'm happy to rework it that way.

What changes

• cli-eval-schedule.yml is new and owned by @supabase/cli. It dispatches eval-refresh.yml with suite: cli and an explicit experiment list:
  • Daily, 04:20 UTC: codex-gpt-6-luna-cli-{pinned,stable,beta,next,nodaemon,absent}. Pinned is the control, and running beta and stable in the same run gives the regression rule its same-day comparison.
  • Weekly, Monday 01:20 UTC: claude-code-sonnet-5.5, claude-code-opus-5.5, codex-gpt-6.1-sol, opencode-kimi-k3 and codex-gpt-6-luna, each on stable and next. Luna is included so the whole matrix resolves the same CLI versions in one run.
• Eight new experiments in experiments/cli/ (<model>-cli-{stable,next}), built from the existing presets.
• eval-refresh.yml:
  • The shared nightly now runs regression only.
  • A new results_pr dispatch input (default on) lets a caller skip the results PR. With it off, the Braintrust upload is the only record, so an upload failure fails the job instead of being ignored.
• Each scheduled run records cadence: daily or cadence: weekly on its Braintrust experiments, through a new cadence dispatch input on eval-refresh.yml. That lets Braintrust and the Hex dashboard tell the two schedules apart without inferring it from the experiment list.
• Both cadences dispatch with results_pr: false, so CLI results go to the Evals project in Braintrust only. Nothing opens or merges a cli-eval-results.json PR.
• CODEOWNERS and the CLI section of CONTRIBUTING are updated to match.

I went with a dispatch rather than a second cron in eval-refresh.yml, for three reasons:

• there's still one run and publish pipeline
• we can adjust our own times and model list without adding churn to your workflow
• the CLI suite gets its own 60-minute run-evals budget instead of sharing one with regression

Testing

• pnpm format:check, pnpm typecheck and pnpm test pass.
• I simulated eval-refresh's pair discovery for both lists: 148 pairs daily and 260 weekly. Every experiment matches at least one eval.
• The dispatch path itself only runs once this is on main. I'm happy to dispatch the weekly list against this branch first to time it, if you'd like that before merging.

Worth a look

• ⚠️ Every run on main shares the eval-refresh-refs/heads/main concurrency group, which allows one running and one pending. The CLI slots are spaced away from the 06:15 nightly so they shouldn't stack. A manual dispatch on main in the same window could still bump a pending run.
• ⚠️ The weekly run is 780 sandbox runs (260 pairs × 3) against the 60-minute run-evals timeout, and I haven't timed it yet. If it runs over, the job fails and only the pairs that completed reach Braintrust.
• ⚠️ The matrix experiments live in experiments/cli/, so run-evals or run-evals-changed on a CLI eval PR now runs 14 arms instead of 6.
• ⚠️ Since the schedule no longer updates cli-eval-results.json, the CLI page on the evals site and the cli-results.jsonl history on GitHub Pages stop moving (the site's Braintrust fetch only covers the benchmark suite). PR runs with run-evals labels still commit it to their branch. We're reading CLI results from Braintrust in Hex now.
• The model names differ from the original proposal (sonnet-5, gpt-6-sol), because the presets have since moved to sonnet-5.5 and gpt-6.1-sol.

Closes CLI-2714
Closes CLI-2715
Closes CLI-2927

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
evals	Ignored	Preview	Nov 4, 2026 2:20pm UTC