test(cli): raise the tool-profile mutation floors to the scores they reach - #854
Merged
Merged
Conversation
…reach With a 20 s timeout (#837) a slow run no longer counts as a kill, and with load failures counted (#851) a static mutant that breaks an import is no longer a false survivor. The tool-profile scores are now a property of the tests, so the two or three points of margin under them protected nothing. On #852's CI run: tools-claude 94.5, tools-cursor 97.3, tools-copilot 95.8. Each floor moves to the integer part: 92 -> 94, 94 -> 97, 93 -> 95. Locally, tools-cursor with --force under a throwaway HOME scores 97.3: at a floor of 98 the gate fails ("mutation score 97.3 is below the 98 declared in mutation-scopes.json"), at 97 it passes. Fixes #853 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011x4ms5qcGuZgYhCxfdHMUb AIDD-Session-Id: 4acc9a1c-19bc-4468-b8b6-e86644bcba60
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🎯 What & why
The tool-profile mutation floors sat two or three points under their scores (#853). That margin was there because scores moved with machine load: a slow run timed out and was scored as a kill. Since #837 (
timeoutMS: 20000) and #851 (a test file that fails to load counts as a failed test), the score is a property of the tests, so the margin only let a loss of detection pass unnoticed.🛠️ How it works
cli/mutation-scopes.json, each floor at the integer part of the score #852's CI measured:🧪 How to verify
node scripts/run-mutation.mjs tools-cursor --forceunder a throwaway HOME scores 97.3. At a floor of 98 the gate fails withmutation score 97.3 is below the 98 declared in mutation-scopes.json; at 97 it passes.mutation-scopes.jsonis a harness file, so CI runs every mutation scope on this PR against the new floors, tools-claude and tools-copilot included.🔗 Linked issue
Closes #853
✅ I certify
🤖 Generated with Claude Code
https://claude.ai/code/session_011x4ms5qcGuZgYhCxfdHMUb