Track harness versions independently of experiment configuration - #860
Merged
Conversation
Persist the evaluated harness version through artifacts and aggregation, support pinned PR Review revision overrides, and surface versions in summaries and leaderboards. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Leave dashboards unchanged, display versions without links, and follow the existing first-result summary convention. Retain only concise capability documentation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Resolve the install-agent-harnesses conflict by keeping the engine-sha indirection while adopting main's 1.39.6 default pin (ecf8e31). The pin now lives in the validation step's env block, so the conflict only surfaced on ref: and would otherwise have silently reverted the bump. Document engine-sha in docs/code-review.md as a convenience for one-off revision comparisons that does not require re-pinning or a release. Remove test_pr_review_severity_override_is_only_available_for_smoke_tests (asserted on Rich-colorized help text, which ANSI escapes split in CI) and test_evaluation_does_not_start_when_version_resolution_fails (exited on --company validation for anyone with the documented .env). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: b127f86c-61b7-45da-b427-ce69c320d848
An engine-sha override runs a BC-ALAgents revision other than the reviewed default pin, so it must not reach Braintrust/Kusto or the leaderboard. With no version column on the dashboards, such a run would otherwise render as a row indistinguishable from the pinned default. Gate it through the existing mock input, alongside the modified-only case. Scoring, the job summary, and the recorded agent_version are unaffected, and requeued repeats stay available so an override can be measured over several runs. Promotion remains a default-pin bump plus a BC-Bench release. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: b127f86c-61b7-45da-b427-ce69c320d848
A harness-version experiment is a configuration experiment that changes a pin instead of config.yaml, so it follows the same process: a branch, the default pin update, a version bump, and a draft PR that records what is being evaluated and why. Dispatching from that branch is what the existing git-ref tracking and the leaderboard-branch merge gate are built around. The engine-sha input skips all of that, so state plainly that it is a shortcut for a quick look and not a substitute. Nothing records the intent behind an override, which is why its results are never published. Generalize experiment step 1 beyond config.yaml, note the Use workflow from selector, list the BC PR Review workflow, and add a harness-version row to the experiment PR template. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: b127f86c-61b7-45da-b427-ce69c320d848
Move BC PR Review specifics (engine-sha mechanics, fixed-variable list, --engine-path vs. run pr-review) into docs/code-review.md, leaving only the framework-level rule in EXPERIMENT.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: b127f86c-61b7-45da-b427-ce69c320d848
Sun Haoran (haoranpb)
requested review from
dayland and
Wenjie Fan (gggdttt)
and removed request for
Wenjie Fan (gggdttt)
September 8, 2026 07:13
Wenjie Fan (gggdttt)
approved these changes
Sep 8, 2026
dayland
approved these changes
Sep 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Record a generic
agent_versionindependently ofExperimentConfiguration, so agent harness (e.g. BC PR Review engine) can be compared against the same benchmark without aggregating different versions together.Why
PR Review engine is a special harness comparing to Copilot CLI, the experimentation happens in a difference repo and we only gets a SHA in BC-Bench, so
ExperimentConfigurationis alwaysNone.It is unrealistic to require the team to always bump the benchmark version when doing an experiment, instead, we should offer experimenting with harnesses' version.
Extra convenience
Noticed that team was considering an easier way to experiment with a private branch with PR Review Agent in #849 , we provide a convenient way to do that now as input to the workflow.
Note: providing the SHA as the workflow input is designed for convenience only, because no description is provided for those runs. To experiment with persisted result, follow the Experiment