Skip to content

docs: require a scenario for every detection rule #140

Description

@sepehr-safari

Three of the 16 detection rules have no scenario exercising them:
TIMEOUT_NO_HEARTBEAT, FIRMWARE_UPDATE_FAILURE and
REPEATED_BOOT_NOTIFICATION. A rule with no scenario has no regression guard,
so nothing in the corpus would catch it breaking.

REPEATED_BOOT_NOTIFICATION shows how the gap opens. It shipped in #114 and no
scenario followed, because nothing in the contributor guide asks for one. The
"How to Add a Detection Rule" checklist ends by auditing existing scenarios for
false positives, which is the negative half of the coverage question. The
positive half, a scenario whose expectedFailures contains the new code, is
never requested.

Fix

State the invariant in the "Contributing Detection Rules" section of
CONTRIBUTING.md: every detection rule ships with a scenario that exercises it,
added in the same pull request as the rule.

Add the corresponding step to the "How to Add a Detection Rule" list, next to the
existing false-positive audit, so both directions of coverage are asked for in
the same place.

Why write it down

  • It keeps the corpus honest as rules are added. The scenario count is a v1.0
    exit criterion (ROADMAP.md line 157), and rule coverage is what makes that
    count mean something.
  • It turns the gap into a well-specified newcomer task automatically. Whenever a
    rule lands without a scenario, the follow-up is a good-first-issue that writes
    itself.
  • It is the rule that would have prevented the three issues above from being
    needed at all.

Documentation only. No changeset: CONTRIBUTING.md is not in the package files
list, so it does not ship to npm.

Related

The three uncovered rules are tracked in #137, #138 and #139. This issue is the
process change that stops a fourth appearing.

Activity

  1. added 2 commits that reference this issue on Jul 30, 2026
    ab968f0
    1993930
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    package:repoRepository-level changespriority:mediumShould be done in current milestonetype:docsDocumentation improvements

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions