test(cli): a canary mutation scope that fails if Stryker stops counting a load failure as a kill - #857
Merged
Merged
Conversation
…ng a load failure as a kill Stryker's vitest runner scores a mutant that stops a module from loading as survived, on its own and in every release up to 10.0.0 (stryker-js #6150); the reporter from #851 is what turns it into a kill. Nothing in the suite proved the last step, Stryker itself scoring that mutant Killed. tests/fixtures/stryker-canary/settings.ts holds exactly one mutant, a static StringLiteral whose replacement makes JSON.parse throw on import. The canary scope mutates it with a floor of 100. It runs whenever the harness changes, so on every Stryker or vitest upgrade, and on any change to the reporter. Evidence: - inventory: one mutant, Killed, "Unexpected end of JSON input"; - reporter neutralized: Survived with testsCompleted 0, and the gate fails (exit 1, "mutation score 0.0 is below the 100 declared in mutation-scopes.json"); - restored: score 100.0, floor 100. The architecture check that every scope matches a file now also looks under tests/fixtures, the one place a scope may mutate something that is not product. Fixes #856 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011x4ms5qcGuZgYhCxfdHMUb AIDD-Session-Id: 4acc9a1c-19bc-4468-b8b6-e86644bcba60
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🎯 What & why
Stryker's vitest runner scores a mutant that stops a module from loading as survived, in every release up to 10.0.0. Upstream tracks it as stryker-js#6150, where the 10.0.0 repro is now posted. No Stryker option or runner option changes it. #851 fixed it here with a vitest reporter, and the suite proves what a real vitest run hands Stryker's collection. Nothing proved the last step: Stryker itself scoring that mutant Killed (#856). So a Stryker or vitest upgrade that bypassed the reporter would bring the false survivors back silently.
🛠️ How it works
cli/tests/fixtures/stryker-canary/settings.tsholds exactly one mutant: a staticStringLiteralwhose replacement makesJSON.parsethrow on import. Its unit test checks the unmutated value and runs in every ordinary suite.mutation-scopes.jsondeclares acanaryscope over it, with a floor of 100 and the reason in$comment;package.jsongetstest:mutation:canary.mutation-scopes-to-run.mjsruns it on a change to the harness (package.json, the lockfile, the Stryker config), totests/helpers/or to the fixture, and not otherwise.mutation-covers-source.arch.test.ts: "every scope matches a file" now also looks undertests/fixtures, the one place a scope may mutate something that is not product.🧪 How to verify
stryker run --mutate tests/fixtures/stryker-canary/settings.tsgives one mutant,Killed,Unexpected end of JSON input.SurvivedwithtestsCompleted0. Through the gate,node scripts/run-mutation.mjs canary --forceexits 1 withmutation score 0.0 is below the 100 declared in mutation-scopes.json.Report: reports/mutation/canary/ (score 100.0, floor 100).scope "canary" ... matches no file. Green after.CHANGED=cli/package.jsonandCHANGED=cli/tests/helpers/unloadable-file-as-failed-test.tsselectcanary;CHANGED=cli/src/kernel/paths.tsgives["kernel"].tests/helpers/unloadable-file-as-failed-test.tsand keep the canary.🔗 Linked issue
Closes #856
✅ I certify
🤖 Generated with Claude Code
https://claude.ai/code/session_011x4ms5qcGuZgYhCxfdHMUb