Add advanced code review metrics view - #851
Conversation
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 57308020-6803-4e62-b08e-57fa875bff89
|
Marking this draft while the advanced data contract is completed. BC-ALAgents currently logs BCQuality filtering/consumption but does not persist article retained, pruned, cited/used, sub-skill execution, or usage-completeness diagnostics into BC-Bench result artifacts, so this page cannot yet show the requested full performance set. |
Restore raw usage and BCQuality diagnostics to result data, aggregate them with explicit telemetry coverage, and expose them in the advanced code-review view. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 57308020-6803-4e62-b08e-57fa875bff89
Include immutable BC-Bench, Copilot CLI, BC-ALAgents, and BCQuality identity in summaries and aggregate grouping, and display linked pins on both code-review dashboards. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 57308020-6803-4e62-b08e-57fa875bff89
Track the current carry-over-focused IsHandled article name so dataset coverage remains aligned with BCQuality main. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 57308020-6803-4e62-b08e-57fa875bff89
Keep historical direct-agent results available while making BC PR Review the baseline and performance comparison surface. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 57308020-6803-4e62-b08e-57fa875bff89
Sun Haoran (haoranpb)
left a comment
Sun Haoran (haoranpb)
left a comment
There was a problem hiding this comment.
dayland I think lots of the changes here is not needed, if we strictly follow the versioning policy of BC-Bench.
A version of BC-Bench, e.g. v0.10.0, specifically list the version of all agent harness.
So a fixed version of BC-Bench should point to a fixed version of all agent harnesses, which should then points to fixed versions of bcquality etc
Note: the version of BC-Bench is already collected as part of the results
Replace the partial Luna and Gemini baseline with three production runs for all eight curated models, preserving legacy direct-agent history. Backfill the immutable BC-Bench, Copilot CLI, BC-ALAgents, and BCQuality identity used by the refresh. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 57308020-6803-4e62-b08e-57fa875bff89
Retain the complete PR Review harness identity coverage while incorporating the AL tool runtime compatibility changes from main. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 57308020-6803-4e62-b08e-57fa875bff89
Keep the advanced metrics and harness identity implementation isolated from the production baseline data update. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 57308020-6803-4e62-b08e-57fa875bff89
|
Sun Haoran (@haoranpb) That makes sense as the intended versioning contract. Would you be open to a thin compromise where we keep the exact pins in the persisted result identity and aggregation key, but only show them compactly as an Evaluation Stack cell on the dashboard? The release remains the canonical description, while the abbreviated BC-Bench, BC-ALAgents, and BCQuality identities save analysts from reconstructing the transitive pins for each row. This also prevents accidental aggregation if results are produced from different commits or experimental pins under the same in-development benchmark version. |
Persist the evaluated harness version through artifacts and aggregation, support pinned PR Review revision overrides, and surface versions in summaries and leaderboards. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Leave dashboards unchanged, display versions without links, and follow the existing first-result summary convention. Retain only concise capability documentation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Resolve the install-agent-harnesses conflict by keeping the engine-sha indirection while adopting main's 1.39.6 default pin (ecf8e31). The pin now lives in the validation step's env block, so the conflict only surfaced on ref: and would otherwise have silently reverted the bump. Document engine-sha in docs/code-review.md as a convenience for one-off revision comparisons that does not require re-pinning or a release. Remove test_pr_review_severity_override_is_only_available_for_smoke_tests (asserted on Rich-colorized help text, which ANSI escapes split in CI) and test_evaluation_does_not_start_when_version_resolution_fails (exited on --company validation for anyone with the documented .env). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: b127f86c-61b7-45da-b427-ce69c320d848
An engine-sha override runs a BC-ALAgents revision other than the reviewed default pin, so it must not reach Braintrust/Kusto or the leaderboard. With no version column on the dashboards, such a run would otherwise render as a row indistinguishable from the pinned default. Gate it through the existing mock input, alongside the modified-only case. Scoring, the job summary, and the recorded agent_version are unaffected, and requeued repeats stay available so an override can be measured over several runs. Promotion remains a default-pin bump plus a BC-Bench release. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: b127f86c-61b7-45da-b427-ce69c320d848
A harness-version experiment is a configuration experiment that changes a pin instead of config.yaml, so it follows the same process: a branch, the default pin update, a version bump, and a draft PR that records what is being evaluated and why. Dispatching from that branch is what the existing git-ref tracking and the leaderboard-branch merge gate are built around. The engine-sha input skips all of that, so state plainly that it is a shortcut for a quick look and not a substitute. Nothing records the intent behind an override, which is why its results are never published. Generalize experiment step 1 beyond config.yaml, note the Use workflow from selector, list the BC PR Review workflow, and add a harness-version row to the experiment PR template. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: b127f86c-61b7-45da-b427-ce69c320d848
Move BC PR Review specifics (engine-sha mechanics, fixed-variable list, --engine-path vs. run pr-review) into docs/code-review.md, leaving only the framework-level rule in EXPERIMENT.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: b127f86c-61b7-45da-b427-ce69c320d848
dayland Good point, you pointed out a very valid gap in the current design:
We never had that need before PR Review Agent, after some thought, what do you think about this #860 ? Could it be an alternative to what we are trying to achieve here? Or at least partly |
Use the generic agent_version field for BC-ALAgents, retain transitive BCQuality provenance, and pin the merged diagnostics support. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 57308020-6803-4e62-b08e-57fa875bff89
Keep transitive PR Review provenance on the advanced metrics page while using the shared benchmark version presentation on default leaderboards. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 57308020-6803-4e62-b08e-57fa875bff89
There was a problem hiding this comment.
🟡 Changes recommended
A few implementation details make the new “advanced diagnostics” metrics potentially inaccurate or brittle (notably knowledge-usage counting, per-task BCQuality scanning cost, and hardcoded Copilot CLI provenance output).
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR extends the code-review (PR Review) evaluation pipeline to persist and surface richer diagnostics (usage token breakdowns, API call stats, BCQuality knowledge/sub-skill telemetry) and adds an advanced “Code Review Advanced Metrics” dashboard to display the expanded schema, while keeping the default leaderboard view focused on headline metrics.
Changes:
- Persist PR Review diagnostics into result / summary / leaderboard schemas and aggregate them with “only where available” semantics.
- Record evaluation stack provenance (BC-Bench commit, Copilot CLI version, BCQuality repo/commit/version) end-to-end through workflows and summaries.
- Add a dedicated advanced metrics documentation page and adjust the default Code Review dashboard to emphasize production PR Review runs vs legacy direct-agent rows.
File summaries
| File | Description |
|---|---|
| tests/test_review_workflows.py | Updates workflow/action pin expectations and adds docs/dashboard visibility assertions. |
| tests/test_pr_review_metrics.py | Expands PR Review metrics tests for new diagnostics and knowledge/sub-skill counting. |
| tests/test_pr_review_metrics_reporting.py | Verifies new diagnostics flow through result JSON, summaries, and aggregates. |
| tests/test_evaluation_summary.py | Adds coverage for provenance fields sourced from environment variables. |
| tests/test_agent_version_results.py | Ensures legacy bc_alagents_commit is restored into agent_version. |
| src/bcbench/types.py | Extends AgentMetrics with PR Review diagnostic fields and validation constraints. |
| src/bcbench/results/summary.py | Adds provenance fields to summaries and restores legacy PR Review agent identity. |
| src/bcbench/results/leaderboard.py | Adds provenance fields to aggregates and restores legacy PR Review agent identity. |
| src/bcbench/results/codereview.py | Computes/serializes new diagnostic averages and coverage rates for code-review summaries. |
| src/bcbench/agent/pr_review/metrics.py | Implements parsing/derivation of PR Review usage + BCQuality/filter/findings diagnostics. |
| src/bcbench/agent/pr_review/agent.py | Passes bcquality_root into metrics builder to support knowledge diagnostics. |
| docs/code-review.md | Reframes default dashboard into production vs legacy sections and links to advanced view. |
| docs/code-review-details.md | Adds the advanced metrics dashboard with expanded columns and provenance display. |
| .github/workflows/summarize-results.yml | Plumbs provenance inputs into summarization via environment variables. |
| .github/workflows/pr-review-evaluation.yml | Exports harness identity outputs and forwards them into summarization inputs. |
| .github/actions/install-agent-harnesses/action.yml | Adds harness identity outputs (Copilot CLI + BCQuality repo/commit/version). |
Review details
- Files reviewed: 16/16 changed files
- Comments generated: 4
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| def _load_filter_report(path: Path) -> _FilterReport: | ||
| if not path.exists(): | ||
| raise AgentError(f"BCQuality filter report not found at {path}.") | ||
| try: |
There was a problem hiding this comment.
Addressed in ea89f12. A missing _filter-report.json now leaves knowledge_pruned unavailable (null) without failing the evaluation; malformed reports still raise an explicit error.
|
|
||
| def _count_available_knowledge(bcquality_root: Path) -> int: | ||
| def is_knowledge_file(path: Path) -> bool: | ||
| parts = path.relative_to(bcquality_root).parts | ||
| return len(parts) >= 3 and parts[0].lower() in _KNOWLEDGE_LAYERS and parts[1].lower() == "knowledge" | ||
|
|
||
| return sum(1 for path in bcquality_root.rglob("*.md") if path.is_file() and is_knowledge_file(path)) |
There was a problem hiding this comment.
Addressed in ea89f12. The BCQuality knowledge inventory count is cached by resolved checkout path for the process lifetime, with regression coverage.
| cited: set[str] = set() | ||
| for collection_name in ("findings", "subResults"): | ||
| collection = payload.get(collection_name, []) | ||
| if not isinstance(collection, list): | ||
| continue | ||
| for item in collection: | ||
| if not isinstance(item, dict): | ||
| continue | ||
| references = item.get("references", []) | ||
| if not isinstance(references, list): | ||
| continue | ||
| for reference in references: | ||
| if isinstance(reference, dict) and isinstance(reference.get("path"), str) and reference["path"]: | ||
| cited.add(reference["path"].replace("\\", "/").lower()) | ||
|
|
There was a problem hiding this comment.
Addressed in ea89f12. References are normalized and counted only when they resolve to a <layer>/knowledge/**/*.md path; skill/non-knowledge references and traversal paths are excluded.
| $repo = $repo -replace '^https://github.com/', '' -replace '\.git$', '' | ||
|
|
||
| "copilot-cli-version=1.0.82" >> $env:GITHUB_OUTPUT | ||
| "bcquality-repository=$repo" >> $env:GITHUB_OUTPUT | ||
| "bcquality-commit=$commit" >> $env:GITHUB_OUTPUT | ||
| "bcquality-version=$version" >> $env:GITHUB_OUTPUT |
There was a problem hiding this comment.
Addressed in ea89f12 by removing the action provenance outputs entirely. The Copilot CLI version now comes from the engine _run-metrics.json, and BCQuality identity is resolved from the engine config plus the actual prepared checkout commit.
There was a problem hiding this comment.
With #860 merged,can we trim this to persisting and displaying the required diagnostics? benchmark_version and agent_version already identify the setup, so I’m not sure we need five extra inputs on the shared summarization workflow.
I understand the convenience of showing the BCQuality version, but the recorded BC-ALAgents SHA already pins it. Could we resolve bcquality-commit in the notebooks for deeper analysis instead of passing it through the evaluation workflows?
Or, alternatively, collect and resolve the dependency chain as part of the evaluation result
Make optional diagnostics resilient, derive provenance from runtime artifacts, and remove PR Review-specific inputs from the shared summarization workflow. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 57308020-6803-4e62-b08e-57fa875bff89
|
Addressed the latest review feedback in |
Adds a dedicated advanced Code Review metrics view and restores detailed PR Review diagnostics to the result and leaderboard schemas.
With #860 merged,
benchmark_versionandagent_versionremain the canonical benchmark and harness identity fields, andengine-sharemains the workflow override for unpublished comparison runs. This PR adds the PR Review-specific analysis layer:_run-metrics.jsonand resolves BCQuality repository/commit/content version from the engine config and prepared checkout directly into PR Review result artifactsbc_alagents_commitfield toagent_versionwhen loading already-published PR Review rowsThe default PR Review pin moves to BC-ALAgents
fdc02d7020632795810057500d62cff2a61513d7, which includes the merged microsoft/BC-ALAgents#62 change required to persist sub-results for complete article and executed-sub-skill diagnostics.