Skip to content

test(cli): a canary mutation scope that fails if Stryker stops counting a load failure as a kill - #857

Merged
blafourcade merged 1 commit into
nextfrom
test/stryker-canary-scope
Sep 11, 2026
Merged

test(cli): a canary mutation scope that fails if Stryker stops counting a load failure as a kill#857
blafourcade merged 1 commit into
nextfrom
test/stryker-canary-scope

Conversation

@blafourcade

Copy link
Copy Markdown
Contributor

🎯 What & why

Stryker's vitest runner scores a mutant that stops a module from loading as survived, in every release up to 10.0.0. Upstream tracks it as stryker-js#6150, where the 10.0.0 repro is now posted. No Stryker option or runner option changes it. #851 fixed it here with a vitest reporter, and the suite proves what a real vitest run hands Stryker's collection. Nothing proved the last step: Stryker itself scoring that mutant Killed (#856). So a Stryker or vitest upgrade that bypassed the reporter would bring the false survivors back silently.

🛠️ How it works

  • cli/tests/fixtures/stryker-canary/settings.ts holds exactly one mutant: a static StringLiteral whose replacement makes JSON.parse throw on import. Its unit test checks the unmutated value and runs in every ordinary suite.
  • mutation-scopes.json declares a canary scope over it, with a floor of 100 and the reason in $comment; package.json gets test:mutation:canary.
  • mutation-scopes-to-run.mjs runs it on a change to the harness (package.json, the lockfile, the Stryker config), to tests/helpers/ or to the fixture, and not otherwise.
  • mutation-covers-source.arch.test.ts: "every scope matches a file" now also looks under tests/fixtures, the one place a scope may mutate something that is not product.

🧪 How to verify

  • Inventory: stryker run --mutate tests/fixtures/stryker-canary/settings.ts gives one mutant, Killed, Unexpected end of JSON input.
  • Reporter neutralized: Survived with testsCompleted 0. Through the gate, node scripts/run-mutation.mjs canary --force exits 1 with mutation score 0.0 is below the 100 declared in mutation-scopes.json.
  • Restored: Report: reports/mutation/canary/ (score 100.0, floor 100).
  • Arch red first: with the scope added and the check unchanged, scope "canary" ... matches no file. Green after.
  • CHANGED=cli/package.json and CHANGED=cli/tests/helpers/unloadable-file-as-failed-test.ts select canary; CHANGED=cli/src/kernel/paths.ts gives ["kernel"].
  • Architecture + fixture test 25 files, 131 tests; typecheck, lint, knip green; scripts suite green.

⚠️ Heads-up

  • One more mutation job, on harness changes only. The scope's single static mutant only reruns the fixture's own test.
  • When stryker-js#6150 is fixed upstream, the canary stays green without the reporter. That is the signal to delete tests/helpers/unloadable-file-as-failed-test.ts and keep the canary.

🔗 Linked issue

Closes #856

✅ I certify

  • I DO CERTIFY I READ EACH LINE OF THE PULL REQUEST BECAUSE I AM A SOFTWARE ENGINEER, NOT A AI PUPPY.

🤖 Generated with Claude Code

https://claude.ai/code/session_011x4ms5qcGuZgYhCxfdHMUb

…ng a load failure as a kill

Stryker's vitest runner scores a mutant that stops a module from loading as
survived, on its own and in every release up to 10.0.0 (stryker-js #6150);
the reporter from #851 is what turns it into a kill. Nothing in the suite
proved the last step, Stryker itself scoring that mutant Killed.

tests/fixtures/stryker-canary/settings.ts holds exactly one mutant, a static
StringLiteral whose replacement makes JSON.parse throw on import. The canary
scope mutates it with a floor of 100. It runs whenever the harness changes,
so on every Stryker or vitest upgrade, and on any change to the reporter.

Evidence:
- inventory: one mutant, Killed, "Unexpected end of JSON input";
- reporter neutralized: Survived with testsCompleted 0, and the gate fails
  (exit 1, "mutation score 0.0 is below the 100 declared in mutation-scopes.json");
- restored: score 100.0, floor 100.

The architecture check that every scope matches a file now also looks under
tests/fixtures, the one place a scope may mutate something that is not product.

Fixes #856

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011x4ms5qcGuZgYhCxfdHMUb
AIDD-Session-Id: 4acc9a1c-19bc-4468-b8b6-e86644bcba60
@blafourcade
blafourcade requested a review from a team as a code owner September 11, 2026 07:16
@blafourcade
blafourcade merged commit 596967b into next Sep 11, 2026
37 checks passed
@blafourcade
blafourcade deleted the test/stryker-canary-scope branch September 11, 2026 07:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant