feat(framework): validate the framework's files deterministically #1149

Problem

The framework's structure is held by rules a model reads, not by checks. PR #601 said it: "R6, R7, R8, R11, R12, R13, R14, R15, R16 and R18 are all testable — […] but nothing runs them in CI." So a skill drifts, a description gets truncated, and each drift costs tokens in every session. The generator's review catches some of it, with a model, after the fact.

Spotify's Shunt moved the same idea from advice to enforcement: "The rules were advisory, not enforced. Claude could ignore them." Its hook measures, then answers.

Measured on next, 2026-11-05:

Scope

One validator, deterministic: no model, no network, same input gives the same output.

The limits above are proposals, to set from the measures; the first run records what already exceeds them.

Configuration: every limit and name pattern lives in JSON config, mapped to the files it applies to; no check hardcodes one.

Example config, an excerpt. Limits and section order are proposals:

{
  "kinds": {
    "skill-folder": {
      "match": ["plugins/*/skills/*/"],
      "rules": { "V8": { "maxLength": 24, "pattern": "^\\d{2}-[a-z0-9]+(-[a-z0-9]+)*$" } }
    },
    "skill": {
      "match": ["plugins/*/skills/*/SKILL.md"],
      "rules": {
        "V1": { "required": ["name", "description"], "nameEquals": "parent" },
        "V2": { "maxChars": 240, "maxTotalChars": 12000 },
        "V5": { "maxLines": 70 }
      }
    },
    "action": {
      "match": ["plugins/*/skills/*/actions/**/*.md"],
      "rules": {
        "V3": { "required": ["Output", "Process"], "forbidden": ["Test", "Tests"], "order": ["Input", "Output", "Process", "Rules"] },
        "V4": { "section": "Process" },
        "V5": { "maxLines": 70 },
        "V8": { "maxLength": 32, "pattern": "^\\d{2}-[a-z0-9]+(-[a-z0-9]+)*\\.md$" },
        "V9": { "forbidden": [{ "text": "aidd_docs/memory/", "why": "memory is already loaded, state the rule without the path" }] },
        "V10": { "forms": ["markdown", "include", "src"] }
      }
    },
    "reference": {
      "match": ["plugins/*/skills/*/references/**"],
      "rules": {
        "V5": { "maxLines": 125 },
        "V8": { "maxLength": 36, "pattern": "^(\\d{2}-)?[a-z0-9]+(-[a-z0-9]+)*\\.(md|json)$" }
      }
    },
    "agent": {
      "match": ["plugins/*/agents/*.md"],
      "rules": {
        "V1": { "required": ["name", "description"], "nameEquals": "stem" },
        "V8": { "maxLength": 16, "pattern": "^[a-z]+(-[a-z0-9]+)*\\.md$" }
      }
    }
  }
}

Example report, from a prototype run on next with a config like this one:

plugins/aidd-vcs/skills/05-resolve-conflict/SKILL.md:2 V1 name "resolve-conflict" differs from "05-resolve-conflict"
plugins/aidd-refine/skills/05-review-conversation/SKILL.md:3 V2 description is 354 characters, limit 240: open on the distinct intent, cut the rest
plugins/aidd-dev/skills/05-review/actions/02-review.md:19 V3 "## Output" after "## Process": order is Input, Output, Process, Rules
plugins/aidd-context/skills/00-onboard/actions/01-scan.md:22 V3 "## Test" is forbidden: move the invariant to Rules
plugins/aidd-dev/skills/09-for-sure/actions/02-auto-accept.md:15 V4 not a numbered step: "Apply these rules in order to "
plugins/aidd-orchestrator/skills/00-async-dev/SKILL.md:1 V5 131 lines, limit 70: move detail to a reference
plugins/aidd-telemetry/skills/01-usage/actions/frame.md V8 name does not match ^\d{2}-[a-z0-9]+(-[a-z0-9]+)*\.md$: rename it

Example of a forbidden mention, a real line from the feat/review-comment-faq branch:

- Reach the pull or merge request through the VCS tool `aidd_docs/memory/vcs.md` names, else the one the remote points to.

plugins/aidd-vcs/skills/06-answer-comments/SKILL.md:26 V9 names "aidd_docs/memory/": memory is already loaded, state the rule without the path

Example of a broken config, which stops the run with exit 2 instead of silently disabling a check:

config script: unknown rule "V9"
config agent.V8: unknown key "maxLenght"
config agent.V8.maxLength: expected integer
config hook: glob "plugins/*/hook/**" matches nothing
config plugins/aidd-context/hooks/hooks.json: 0 kinds, expected 1

Expected action shape:

# Plan

## Input
- The request to plan.

## Process
1. Read the request.
2. Write the phases.

## Output
- `plan.md`, one section per phase.

Generation: a generate-skill-action script writes this skeleton, so a new file starts valid and the generator's prose stops describing a shape a script can produce.

Where it runs, one engine for all three:

1. By hand: node scripts/validate-framework.cjs [path...], exit 1 on an error.
2. Write hook: hands the model the lines its own write broke, shaped <file>:<line> <rule> <message>, like #1146's memory hook. It reuses that engine rather than adding a second one.
3. Pre-commit and CI: block.

Existing drift: a baseline lists today's violations, and may only shrink, so the validator can block from day one. An entry that no longer fires fails the run.

Absorbed from the issues this one closes

• #985: remove the Test / Tests sections from every skill, and give constraints one skill-level Rules section plus an optional action-level one. Every meaningful verification step keeps a home. The validator then refuses a ## Test section. The generator's R11 and R13 change with it.
• #794: rewrite the descriptions it lists (aidd-telemetry, 12-cook, 10-todo, 08-three-amigos, 01-sdlc, 07-refactor, 02-challenge, 09-for-sure) so the distinct intent comes first and survives truncation, and align R3 and its template. Its table and acceptance criteria stay the reference. Note: its measures predate today's above.

To brainstorm

To settle before building:

• Mapping: how a config maps rules to files: by file kind, by glob, or both.
• Files: one JSON for everything, or one per family of rules.
• Prefix: which kinds require NN-. Today skills do, references almost never, agents never, and 6 actions don't.
• Size: whether a name's limit covers the file name alone or the whole path (longest today: 95 characters).
• Overrides: whether a plugin can tighten or loosen the shared config.
• Mentions: which ones stay legitimate. On next, 45 lines in 17 files name aidd_docs/memory, 38 of them in aidd-context, which writes that memory; 07-refactor also records into it. A script can't tell reading from writing, so these need an exception per path or the baseline.
• Links: whether to also check paths in backticks, which may be relative to the file, to the plugin or to the user's project.

Acceptance criteria

• make validate runs every check above, and exits 0 on next with the baseline.
• Changing a limit or a name pattern takes a JSON edit, no code.
• Each rule has a fixture that breaks it and a test that sees it fail, so a rule that stops firing fails the suite.
• Two runs on the same tree give byte-identical output; no check reads the network or calls a model.
• Every error names the file, the line where it has one, the rule, and what to do.
• A write breaking a rule gets its error back in the same turn on each host the memory hook covers; the hook stays silent otherwise and never fails the tool call.
• A new skill and action from the generator pass with no edit.
• No shipped skill or action holds a Test section, and the invariants those sections carried have a home.
• The descriptions docs(framework): fiabiliser les descriptions des skills sous troncature Codex #794 lists are rewritten; a plain request for a task list does not select 10-todo.
• aidd-orchestrator's validator agent (#983) runs the validator first, and judges only what a script cannot.

Prior art

• #1146: memory-file linter and write hook, rules M1 to M7 (a prose line over 25 words, a file over 400). The first slice, and the engine to extend.
• #1148: a skill's name equals its folder, held by a test. Moves into the validator.
• #1142: make check-hosts checks what each tool really loads. The installed side of this issue's source side.
• #250: "il doit être déterministe", the pre-commit and CI model for orthogonality.
• aidd#46: a validation script reporting file:line, wired to a blocking lefthook.
• Already mechanical, to keep or fold in: scripts/check-skill-argument-hints.mjs (R4), scripts/lib/architecture-scan.js (router lists every action), lefthook.yml skill-frontmatter (presence only), scripts/check-markdown-links.js.
• Related: #1017 (generator aligned with Anthropic's guidance), #978 (tool compatibility).

Out of scope

• Judging whether a skill is good: that stays with review, and #1079 tests what a skill makes an agent do.
• Shunt's delegation of reads to a cheaper model, which belongs to feat(framework): add AIDD exploration, task validation and configurable delegation #983.
• Third-party plugins' skills.