Skip to content

Experiment: add area playbooks to fix-bug agent - #862

Draft
alexei-dobriansky wants to merge 5 commits into
private/ventselartur/BugFixingSkillToBCBenchfrom
experiment/bugfix-area-playbooks-design
Draft

Experiment: add area playbooks to fix-bug agent#862
alexei-dobriansky wants to merge 5 commits into
private/ventselartur/BugFixingSkillToBCBenchfrom
experiment/bugfix-area-playbooks-design

Conversation

@alexei-dobriansky

@alexei-dobriansky alexei-dobriansky commented Sep 7, 2026

Copy link
Copy Markdown
Collaborator

Stacked dependency

Depends on #845. This PR is based on private/ventselartur/BugFixingSkillToBCBench; review only the commits above that branch.

Experiment

Measures whether area-specific playbooks improve the existing fix-bug custom agent on the same 52-task dataset.

  • Vendors exact Warehouse, Manufacturing, Costing, and Subscription Billing playbooks from microsoft/BCAppsBugFix#356
  • Adds discover mode, where the agent loads at most one playbook after locating a matching source path
  • Adds selected mode, where BC-Bench selects a playbook from an unambiguous metadata.area value before investigation
  • Leaves broad inventory metadata unmapped in selected mode because those tasks span several domains
  • Records playbook mode, content revision, and selected playbook ID in evaluation results
  • Configures this branch for the production-representative discover arm

Direct discovery coverage is 15 of 52 tasks: Warehouse/Tracking 8, Manufacturing 5, Costing 1, and Subscription Billing 1. The dataset, model, tools, bug-fix evaluation pipeline, and scoring remain unchanged from #845.

Hypothesis

Area guidance should improve resolution on directly covered tasks without affecting uncovered tasks. Selected mode is retained as an upper-bound diagnostic if discovery routing underperforms.

Validation

  • Exact playbook content matches microsoft/BCAppsBugFix#356 at 1cc59c631e58dca7997f93a136826733821d8c00
  • uv run pytest -q — 918 passed, 1 skipped, 1 deselected
  • uv run pre-commit run --all-files

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 3dd7665c-7836-4fd9-964b-5fe37e99b462
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 3dd7665c-7836-4fd9-964b-5fe37e99b462
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 3dd7665c-7836-4fd9-964b-5fe37e99b462
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 3dd7665c-7836-4fd9-964b-5fe37e99b462
@alexei-dobriansky alexei-dobriansky changed the title Experiment: add Warehouse playbook to fix-bug agent Experiment: add area playbooks to fix-bug agent Sep 7, 2026
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 3dd7665c-7836-4fd9-964b-5fe37e99b462
@alexei-dobriansky

Copy link
Copy Markdown
Collaborator Author

Follow-up from workflow run 34165334991: commit 0b3d8aa prevents the fix-bug agent from delegating/backgrounding unattended work and persists empty AL diffs as failed benchmark results instead of failing the matrix job. Validation: 921 passed, 1 skipped, 1 deselected; all pre-commit hooks passed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant