Skip to content

test(cli): raise the tool-profile mutation floors to the scores they reach - #854

Merged
blafourcade merged 1 commit into
nextfrom
test/raise-tool-profile-mutation-floors
Sep 11, 2026
Merged

test(cli): raise the tool-profile mutation floors to the scores they reach#854
blafourcade merged 1 commit into
nextfrom
test/raise-tool-profile-mutation-floors

Conversation

@blafourcade

Copy link
Copy Markdown
Contributor

🎯 What & why

The tool-profile mutation floors sat two or three points under their scores (#853). That margin was there because scores moved with machine load: a slow run timed out and was scored as a kill. Since #837 (timeoutMS: 20000) and #851 (a test file that fails to load counts as a failed test), the score is a property of the tests, so the margin only let a loss of detection pass unnoticed.

🛠️ How it works

cli/mutation-scopes.json, each floor at the integer part of the score #852's CI measured:

Scope Score Floor
tools-claude 94.5 92 → 94
tools-cursor 97.3 94 → 97
tools-copilot 95.8 93 → 95

🧪 How to verify

  • Locally, node scripts/run-mutation.mjs tools-cursor --force under a throwaway HOME scores 97.3. At a floor of 98 the gate fails with mutation score 97.3 is below the 98 declared in mutation-scopes.json; at 97 it passes.
  • mutation-scopes.json is a harness file, so CI runs every mutation scope on this PR against the new floors, tools-claude and tools-copilot included.

⚠️ Heads-up

  • No slack left on these three scopes: a change that lets one more mutant survive on a profile fails its gate. That is the point, and it means a new profile field arrives with the test that kills its mutants.
  • The other scopes keep their floors; only these three were re-measured under both fixes.

🔗 Linked issue

Closes #853

✅ I certify

  • I DO CERTIFY I READ EACH LINE OF THE PULL REQUEST BECAUSE I AM A SOFTWARE ENGINEER, NOT A AI PUPPY.

🤖 Generated with Claude Code

https://claude.ai/code/session_011x4ms5qcGuZgYhCxfdHMUb

…reach

With a 20 s timeout (#837) a slow run no longer counts as a kill, and with
load failures counted (#851) a static mutant that breaks an import is no
longer a false survivor. The tool-profile scores are now a property of the
tests, so the two or three points of margin under them protected nothing.

On #852's CI run: tools-claude 94.5, tools-cursor 97.3, tools-copilot 95.8.
Each floor moves to the integer part: 92 -> 94, 94 -> 97, 93 -> 95.

Locally, tools-cursor with --force under a throwaway HOME scores 97.3: at a
floor of 98 the gate fails ("mutation score 97.3 is below the 98 declared in
mutation-scopes.json"), at 97 it passes.

Fixes #853

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011x4ms5qcGuZgYhCxfdHMUb
AIDD-Session-Id: 4acc9a1c-19bc-4468-b8b6-e86644bcba60
@blafourcade
blafourcade requested a review from a team as a code owner September 11, 2026 06:32
@blafourcade
blafourcade merged commit c8e03d0 into next Sep 11, 2026
36 checks passed
@blafourcade
blafourcade deleted the test/raise-tool-profile-mutation-floors branch September 11, 2026 07:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant