Review Anthropic's Code Review Practices Against the Fleet Review Workflow for Cost and Backlog #3391

Activity

1. ptr727

   added

   enhancementNew feature or requestNew feature or request

   agentsAgents instructionsAgents instructions

   on Oct 23, 2026
2. ptr727 commented on Oct 23, 2026

   

   ptr727

   on Oct 23, 2026

   OwnerAuthor

   More actions

   Further context from the maintainer: the same pattern holds in other fleet repositories, not only the hub. Measured as wall time per shipped feature, the return is very low. Most of the time goes to prose and comment work and to waiting on Copilot reviews. Prose quality is improving and may bottom out, letting later work go to features, but at the current cost the funding runs out first. The goal of this issue is therefore cost and backlog first, and review quality second.

   The hub's open backlog at the time of this comment, not counting handoff links:

   • 394 open issues in total.
   • 64 are write guard reads, nearly all shell edge cases in how the guard parses a command.
   • 106 carry the prose, comments, or documentation label.
   • So about 43 percent of the backlog is guard hardening or wording, and most of it was filed by the review loop itself.

   Levers worth weighing here, each a maintainer decision:

   1. A stated threat model for the write guard. The guard backstops mistakes by an agent acting in good faith, and is not an adversarial sandbox. Under that model, a parse gap that only a contrived command reaches gets closed as out of scope rather than fixed, and the guard backlog shrinks to the cases an agent would plausibly type.
   2. One hosted review per change rather than one per pull request. A feature pull request into develop would carry the local pass alone, and Copilot would review once, at the promotion. That cuts both the waiting time and the Copilot cost on every feature pull request.
   3. A severity bar on filing. A review finding below that bar is declined on the pull request rather than filed as an issue, so the review loop stops growing the backlog with wording and style follow-ups.
3. ptr727 commented on Oct 23, 2026

   

   ptr727

   on Oct 23, 2026

   OwnerAuthor

   More actions

   Re-review this in context of #1311 that laid out a plan to address the issue growth.
4. ptr727 commented on Oct 23, 2026

   

   ptr727

   on Oct 23, 2026

   OwnerAuthor

   More actions

   Re-review against #1311: the pump moved from prose to the write guard, and two filing rules set the rate

   Live reads taken 2026-10-23 for 2026-10-16 to 2026-10-23 across ProjectTemplate, Financial-Modeling, HomeAutomation-Config, and ESPHome-NonRoot.

   1. What #1311 and #1631 changed, and what they did not

   • Why the Backlog Grows: Prose Review Has No Fixed Point, and a Plan to Keep the Rigor Without It #1311 shipped 3 of 13 sub-issues: P0: Commit a Stop Condition and Round Budget for the Whole-Unit and Local Review Passes #1312 (stop rule), P0: Decouple the Review Ledger and the Skills Digest From the Working Tree #1313 (ledger decouple), P0: Scope Reviewer Bots Away From Generated Mirrors and Bound What the Merge Gate Owes an Out-of-Diff Prose Finding #1314 (bot scoping). Review rounds are spent debating prose in comments instead of fixing code #1631's decisions landed item 1 (two-round local budget, whole-unit read moved to the weekly sweep) and item 2 (the comment gate, 2.0.639). Items 3 and 4 (PR-title gate, no re-review of a byte-identical promotion, Copilot told not to review Markdown wording) show no evidence of landing. The structural items P1: Define the Normative-Statement Format, Lint It, and Pilot It on "Verification Discipline" #1316 to P2: Add Weekly Backlog and Review-Cost Metrics #1324 never ran.
   • Prose is no longer the filing rate. This week 10 prose and 30 documentation issues were filed fleet-wide out of 329 non-handoff issues. The hub still holds 102 open prose or documentation issues, nearly all from September, frozen as Why the Backlog Grows: Prose Review Has No Fixed Point, and a Plan to Keep the Rigor Without It #1311 said and never closed.
   • The write guard is the new pump. host-setup/agent-safety/claude/gh-write-guard.py grew from 799 lines on 2026-08-01 to 4,650 on 09-01, 9,819 on 10-01, and 21,089 today. October filed 135 "Write Guard" issues, 61 open now, and 41 guard pull requests merged into develop. Nearly every open guard issue opens with "Found by the local review of the change closing #N": each fix's local pass reads a 21k-line hand-written bash parser and finds the next shell shape. That is Why the Backlog Grows: Prose Review Has No Fixed Point, and a Plan to Keep the Rigor Without It #1311's no-fixed-point loop moved from prose to a parser whose input domain is unbounded.
   • The fleet's own rule already forbids this. GOVERNANCE.md "Trust Boundaries and Hardening Effort" says "Do not build a parser for adversarial shapes or test injection through a trusted file", and host-setup/agent-safety/README.md says the hooks "bound an accident rather than an adversary".

   2. This week, four repositories

   Repository	Issues opened	Handoff links	Non-handoff still open	Non-handoff closed	PRs merged	Open now
   ProjectTemplate	182	45	57	122	113	392
   Financial-Modeling	136	35	43	58	113	140
   HomeAutomation-Config	143	62	41	40	120	77
   ESPHome-NonRoot	19	9	1	9	36	2
   Total	480	151	142	229	382	611

   • 151 of 480 issues created (31 percent) are handoff chain links. Each is a create, a comment on the predecessor, and a close.
   • 108 of the 329 non-handoff issues (33 percent) carry pre-existing. That is the local-strict-review rule "a pre-existing one is filed once and blocks nothing" doing exactly what it says, on every pull request.
   • Issues filed per merged pull request: hub 1.2, Financial-Modeling 0.9, HomeAutomation-Config 0.7, against Why the Backlog Grows: Prose Review Has No Fixed Point, and a Plan to Keep the Rigor Without It #1311's target of below 1. Only the hub misses it, and the guard lane is why. The downstream backlogs are mostly features, bugs, and refactors the maintainer wants, so the concern there is the filing rate rather than the kind.

   3. Cost

   	September, full month	October, to the 23rd
   Actions Linux minutes	48,042	57,913, on pace for about 78k
   Copilot gross	$89	$472
   Copilot review runs, this week, four repositories		418
   Workflow runs, this week, four repositories		about 2,200

   The hub used 19,853 of the October minutes, a third of the fleet. #1631 measured agent spend at about $15.4k over two weeks on one machine at list rates, 82 percent of it after a pull request's first review.

   4. Against the Anthropic post

   The post drops style, diagnostics, and cleanup findings before posting, tags a pre-existing defect without filing it and leaves old bugs to a nightly sweep, runs several finders whose findings only post when verifiers agree, caps a pull request at three rounds and then asks the author to reconsider the approach, states its priority as real bugs first and fewer rounds second, tiers human approval by risk, and never re-runs review on identical content.

   The fleet posts every finding and files the pre-existing ones, runs the local pass and each hosted reviewer in series with a written outcome owed per finding, has a two-round local budget but no pull-request cap, requires every finding on head closed at the merge gate, and reviews a feature pull request and again its byte-identical promotion. It also spends where the post would not: a full-file-context adversarial pass per push, a handoff issue per session, and hardening a trusted-input guard against adversarial shells.

   5. Proposed changes, ranked by effect on net open issues and cost per merged pull request

   Each item marked decision is the maintainer's.

   1. Freeze the write guard under its stated threat model (decision). The model exists already: a good-faith agent's mistake, not an adversary. Apply it: a guard issue stays open only where the command shape is one an agent plausibly types in a session. Close the rest of the 61 as out of scope in one sweep, and state the model in one sentence at the top of the guard so the local pass stops filing parser gaps. Consider fail-closed on any command the guard cannot read plainly, which shrinks the code rather than growing it. Effect: about 55 fewer open now, about 15 fewer filed per week, and the hub's largest pull request lane stops.
   2. Stop filing pre-existing findings as issues (decision, rule change in local-strict-review and pr-review-conduct). A pre-existing finding is listed in the pull request body, or appended to one rolling ledger issue per repository, and filed on its own only where it is a correctness defect with a stated failure. Style, consolidation, and wording findings on untouched text are dropped, as the post drops them. Effect: about 100 fewer issues filed per week fleet-wide, and the largest cost-per-pull-request saving, since each filed issue later costs a lane, a pull request, and a review.
   3. Bulk-close the frozen prose backlog (decision). Why the Backlog Grows: Prose Review Has No Fixed Point, and a Plan to Keep the Rigor Without It #1311 froze the prose and procedure issues to close them as superseded when their units were rewritten. The rewrite never ran. Close the 102 with one comment naming this decision, keeping only those that state a claim the shipped code falsifies. Effect: about 90 fewer open hub issues.
   4. One hosted review per change (decision). Copilot at the promotion only, with feature pull requests carrying the local pass and CI, or the reverse with the byte-identical promotion skipped. Effect: Copilot runs roughly halve, Actions minutes fall with them, and promotion wall-clock falls.
   5. Thin the handoff chain (decision). One standing handoff issue per track updated by comment and closed when the track ends, or a link only when a session ends with work unfinished. Effect: about 150 fewer issues created per week and one fewer write per session. The open count does not change, since links close, but every backlog metric stops counting them.
   6. Close the Why the Backlog Grows: Prose Review Has No Fixed Point, and a Plan to Keep the Rigor Without It #1311 structural items that will not run (decision). P1: Define the Normative-Statement Format, Lint It, and Pilot It on "Verification Discipline" #1316 to P2: Add Weekly Backlog and Review-Cost Metrics #1324 have had no activity since September. Pick P2 (decision): Carry Only the Fleet Bootstrap Verbatim, and Reach Everything Else From the Hub #1321, which removes the fleet-wide re-review of every wording change, or close them so the backlog says what is planned.
   7. Add the metric P2: Add Weekly Backlog and Review-Cost Metrics #1324 asked for, cheaply. One script on the existing weekly sweep: non-handoff filed against closed per repository, issues per merged pull request, Copilot runs per merged pull request, Actions minutes per merged pull request.

   Not recommended: more prose rule work, more guard hardening, or deleting the prose rules, which #1631 measured at 2 percent of findings.

   Done when, measured

   • After items 1 to 3, hub open non-handoff issues below about 250 from 387.
   • After item 2 has run a week, pre-existing filings near zero and issues per merged pull request below 0.5 in every repository.
   • After item 4, Copilot review runs per week about half of 418.
   • After item 5, handoff issues created per week under 30.
5. ptr727

   added 8 commits that reference this issue on Oct 23, 2026

   State the Write Guard's Threat Model and Document Two Allowed Shapes (#…

   ...

   6b3a54f

   List Pre-Existing Review Findings in the PR Body Instead of Filing Th…

   ...

   d56248a

   Keep One Standing Handoff Issue per Track (#3402)

   ...

   92875fb

   Run Copilot Review Once per Change at the Promotion (#3403)

   ...

   96d1396

   Name the Request Exception and Reword the Sibling Outcome

   ...

   466adc1

   Align Review Rule Text with Promotion-Only Copilot Review (#3404)

   ...

   d148ffb

   Promote Write Guard Threat Model and Review Cost Changes to Main (#3405)

   ...

   f610e42

   Rewrite the Write Guard as a Small Lexer With Plain-Shape Rules (#3407)

   ...

   946aa4a
6. ptr727 commented on Oct 23, 2026

   

   ptr727

   on Oct 23, 2026

   OwnerAuthor

   More actions

   What landed on 2026-10-23 under the decisions above

   The maintainer approved all seven items, item 4 as Copilot at the promotion only, and extended item 1 to a rewrite of the guard.

   • Item 1. State the Write Guard's Threat Model and Document Two Allowed Shapes #3399 states the guard's threat model in its README and docstring. Rewrite the Write Guard as a Small Lexer With Plain-Shape Rules #3407 rewrites gh-write-guard.py from 21,091 lines to 1,653 with 191 source-tagged cases, fail open on shapes it does not read except the protected-default-branch bypass, and adds a 2,500-line ceiling test. 31 guard issues closed as out of scope, 27 kept as over-denials or shapes a session hit.
   • Item 2. List Pre-Existing Review Findings in the PR Body Instead of Filing Them #3401 rewrites pr-review-conduct outcome 4: a pre-existing finding is listed in the pull request body, filed only as a correctness defect with a stated failure. Align Review Rule Text with Promotion-Only Copilot Review #3404 aligns the Merge Gate text and the class-sweep sibling rule. The nine pull requests driven under the new rule filed zero issues.
   • Item 3. 49 frozen prose and documentation issues closed, 46 kept where a document states something the code does not do.
   • Item 4. Run Copilot Review Once per Change at the Promotion #3403 drops Copilot's automatic review from the develop ruleset config and reads an attested local pass as coverage on any feature pull request. Applied live to the hub and 21 other cataloged repositories through configure.sh apply.
   • Item 5. Keep One Standing Handoff Issue per Track #3402 makes handoff.py keep one standing issue per track, edited in place with the outgoing body archived as a comment, and adds close. Every repository with open handoffs now holds one standing default issue, 30 tracks folded fleet-wide.
   • Item 6. Why the Backlog Grows: Prose Review Has No Fixed Point, and a Plan to Keep the Rigor Without It #1311 and its unworked sub-issues closed, P2 (decision): Carry Only the Fleet Bootstrap Verbatim, and Reach Everything Else From the Hub #1321 kept.
   • Item 7. Add Weekly Backlog and Review-Cost Report Script #3400 adds scripts/backlog_metrics.py. HomeAutomation-Config#1353 carries the weekly mail timer on the automation host, since GitHub Actions has no way to deliver the report.
   • Fleet audit. 226 open issues closed across the hub, Financial-Modeling, HomeAutomation-Config, and twenty small repositories, each with a comment citing this decision. Kept: every bug with a stated failure, every maintainer want, every open decision, and HomeAutomation-Config's refactor chores at the maintainer's choice.
   • Promotion Promote Write Guard Threat Model and Review Cost Changes to Main #3405 merged and released as 2.0.1148.

   Queued under the same decisions, each in its own pull request: the pr_review.py split into gate core, Copilot adapter with JSON vocabulary, and cleanup script; the prose gate on Pygments and markdown-it-py with its vocabulary and exemptions in JSON and a repo-local override; the audit's self-test moved to tests/, PyYAML and markdown-it-py for its reading, and a check registry; and the two CODESTYLE rules that tests live in tests/ and a maintained parser replaces a hand-written one, with uv run --script as the dependency mechanism.
7. ptr727

   added 4 commits that reference this issue on Oct 23, 2026

   Add Test and Parser Rules and Run Hub Scripts Through uv (#3409)

   ...

   dd172fa

   Hold the Merge Gate on Unplaced Review Text Until Acknowledged (#3415)

   ...

   d95bdba

   Move Audit Tests to tests/, Parse YAML and Markdown, Add Check Regist…

   ...

   c66778b

   Promote the Write Guard Rewrite, uv Scripts, and Parser Gates to Main (…

   ...

   6ec0a89