feat(framework): measurement that works on five tools, each proven by a session that ran

[blafourcade]

🎯 What & why

Know what a feature cost — tokens, models, time — across Claude Code, Codex, Copilot, Cursor and OpenCode, by reading the files those tools already write.

Three acts, and they do not need the same things. Allowing measurement and answering what it cost both go through aidd. Recording needs nothing at all: the hooks are plain zero-dependency node, so a session measured today stays readable by a CLI that was not installed when it ran. Nothing leaves the machine unless you point it somewhere yourself.

🛠️ How it works

Three skills: 00-init allows measurement, 01-cost answers what a period or a task consumed, 02-check says whether the chain is actually recording. Each calls aidd telemetry …; none ships a script of its own.

One contract, five adapters. Each tool's file shape maps to one record through a per-tool adapter (claude-code-transcript.ts, codex-rollout.ts, copilot-events.ts, opencode-export.ts), against a closed allowlist in metrics-contract.md. A field not on that list cannot be stored, so no prompt, code or diff ever can be.

An unknown is never a zero. Cursor writes no token count anywhere; Copilot writes a session total and never a per-request figure. Both are reported as what they are, beside the figure, never as a 0 that reads as free. Same rule for attribution: every record states its own strength — tool-stated, journal-interval, unattributed — and an unattributed turn stays its own row instead of being folded into the nearest step.

The sink is append-only. A stored record is never corrected in place, only reconciled on read. That is what lets a session read while still running be superseded later by a strictly better reading of the same turn, rather than frozen partial.

One implementation, not two mirrors. The plugin used to carry its own reporter and its own checker beside the CLI's, kept equal by a parity suite. That second implementation is deleted — 25 files, 4,355 lines — and the parity suite with it. A figure can no longer disagree with itself, because there is only one place that computes it. The hooks stay plain node, because they are the only part that runs per tool call.

The one that mattered. Claude Code writes a line when a message starts and again when it completes, sharing one message.id. The accumulator kept the first — a placeholder carrying output_tokens: 3 where the real figure was 329. Measured across 1,604 real transcripts: 37% of output tokens lost, and no test saw it, because the code was consistent with what it asserted about itself. The fix is validated against Claude Code's own /usage screen over 15,684 billed calls — output and cache-read agree to ~3%; input and cache-write differ by construction, since /usage weights a 1-hour cache line by its price while a transcript counts it once.

🧪 How to verify

cd cli && pnpm test:unit && pnpm test:integration && pnpm test:e2e
node --test "scripts/__tests__/*.test.js"
node scripts/check-cli-layering.mjs && node scripts/check-markdown-links.js

At 4986067d: CLI 3,274 across 298 files, plugin 317 across 19, all green; tsc, biome, knip, layering and link checks clean. cli / Windows installs the packed tarball with npm install -g and then runs aidd --version && aidd telemetry check — a command lifted from 02-check's own markdown, so the job fails if the CLI cannot be resolved on the PATH.

End to end, on a real repository:

aidd telemetry on          # allow measurement for this project
# work as usual, then
aidd telemetry report      # what did it cost
aidd telemetry check       # is the chain actually recording

⚠️ Heads-up

Found by running, not by reading. Each was invisible to a green suite:

• The hook and the CLI shared one string, guarded by nothing. unrecognised_payload is written by the plugin in CommonJS (record.cjs:268) and read by the CLI in TypeScript (telemetry-evidence-adapter.ts:30). Renaming the hook's literal left 11 e2e and 186 plugin tests green: one side asserted only that the marker file existed, the other typed the same literal into its own fixture. With the marker unread, a payload that did arrive reports as "the hook never fired" — an unknown printed as a nothing. Now guarded by a case that spawns the real hook and reads whatever file it writes.
• A directory marker does not survive a rename. The plugin declared "type": "commonjs" through package.json files, and the flat-skill rename moved one away from the scripts it covered. Every CommonJS file it ships is .cjs now, which Node reads as CommonJS whatever the host declares. hooks/opencode-plugin.js is the one exception: OpenCode auto-discovers {plugin,plugins}/*.{ts,js} and nothing else — measured, renaming it .mjs made OpenCode find it, log it, and never run it.
• Cursor never loads a plugin's own hooks.json. Seven declared events, three probes, headless and interactive: not one fired. The project's own .cursor/hooks.json fires normally — the obstacle was where we installed, not Cursor.
• Codex will not run a hook nobody approved, and says nothing. Trust is per entry, so a renamed event inherits no approval, and a headless run never sees the prompt.
• os.homedir() never reads $HOME on Windows, it reads USERPROFILE. The plugin resolved HOME || homedir(), the CLI called bare homedir() — so a Windows run read 43,853 tokens from fixtures a POSIX run read 183,939 from.

One proof can never be re-run. aidd telemetry check was compared verdict-by-verdict against the plugin checker it replaces, on seven scenarios, by restoring the deleted script from git. That checker is gone; this comparison is unreproducible by design.

Known limits are in the plugin's README, each with its measurement. The three that bite first: Codex needs one interactive approval; OpenCode misses a server process's first session, and opencode run is always a first session; only Claude Code's writes name a task.

Not met, under issues in the list below. #703's "a session that resolved no skill is distinguishable from one that needed none" has no implementation. #694's scale bullet was measured against a synthetic tree, not the hundred-session one it names. #631's evidence reads "the same skill proves both"; it is two, 01-cost and 02-check.

#683 closes, with its residual written down rather than hidden. It asked that adding a tool mean writing one file rather than editing five tables in three files. All five are gone as hand-maintained tables: VENDOR_FIELD_BY_HOST, WRITTEN_PATH_EXTRACTOR_BY_HOST and STEP_START_BY_HOST are now Object.fromEntries(Object.entries(TOOLS_BY_HOST)…) views over one table, and SESSION_ID_READER_BY_HOST and CWD_READER_BY_HOST became the readSessionId and readCwd functions in hooks/lib/tools/index.cjs, gone under any name. Each host states its own facts in hooks/lib/tools/<host>.cjs, one module each. The residual is one line: a sixth tool means the module plus its require in index.cjs — two files, not one. host.cjs still names the five as well, but that is payload sniffing, deciding which tool sent this JSON, and no dispatcher shape removes it. The cost the issue was written against — understanding a tool means reading five places — is gone.

Unrelated reorg riding along. 12 files move from aidd_docs/plans/ to aidd_docs/tasks/ (R100 renames, no content change). Nothing here depends on it; say the word and it comes out.

🔗 Linked issue

These do not fire on this merge. GitHub auto-closes only into the default branch, and this targets next. The list is the record of what the branch delivers; the issues need closing by hand at merge time, or when next is promoted.

#617 is already closed by hand — it argued that everything which reads belongs to the plugin, and this branch reverses that on purpose; the reversal is written in its own thread.

Closes #617, #630, #631, #649, #659, #661, #676, #680, #681, #683, #686, #688, #693, #695, #697, #698, #699, #700, #701, #702, #704, #705

#630 closes against #746, which carries what it does not prove. The commit trailer works: eight e2e tests make real commits and read the messages back — an agent's commit carries AIDD-Session-Id, a commit no session made carries nothing, the id is the session actually running rather than one inherited from an environment variable, an amend writes it once, off stops trailering and leaves existing commits alone, and a prepare-commit-msg the repository already had still runs.

Two of its seven boxes are proven by nothing, and both moved to #746 rather than into a closing comment nobody reads. It asks for four host configurations — core.hooksPath, lefthook, husky, none — and only the last is tested; core.hooksPath is handled in code (resolveHooksDir asks git through rev-parse --git-path hooks) and asserted nowhere. And its p95-under-200ms box has no timing test.

What that second reading turned up is worth more than the missing tests. The install appends one line to prepare-commit-msg, and in a repository where another tool generates that file the line is erased the next time it regenerates — measurable on this very repository, where lefthook rewrote commit-msg, pre-commit and pre-push on 2 September and spared prepare-commit-msg only because lefthook.yml declares no job for it. aidd telemetry check carries no claim about the trailer at all, so the loss would be silent. #746 owns the repair — re-establish the call site on session-start, never defend it, so one code path covers a regenerated hook, a hand-made overwrite and a core.hooksPath move without naming a single third-party tool — and the five claims check should make about it.

Every issue in that list was re-verified against the code on 2026-09-02, not inherited from the 2026-08-31 audit — product code moved between the two. What proved each: #659 scripts/probe-identifier-join.cjs prints the attributes it checked and exits 2 for "the probe is broken", behind the cli / Identifier join (Claude Code) job; #661 person-resolution.ts, whose unresolved rows keep the raw identifier that produced them; #680 CURSOR_EVENT_MAP mapping Stop to both stop and sessionEnd; #681 six captured copilot-compat-* fixtures beside the canonical ones; #686 SYNTHETIC_MODEL in claude-code-transcript.ts, with an e2e asserting no <synthetic> reaches a by-model breakdown; #688 one surviving cast, allowlisted in check-cli-layering.mjs with its written reason, and cli/tests now scanned; #693 and #695 worktreeFields in hooks/lib/repo.cjs, absent rather than empty; #697 kind: "session" in copilot-events.ts; #698 acceptsHooks: true on Codex; #699 CODEX_HOOKS_TRUST_NOTICE with its own integration test; #700 commit 2343beea; #701 the captured Copilot skill payloads; #704 the NO_KNOWN_PROJECT symbol; #705 byDays. #631 is the epic, and its one unmet criterion is stated above.

Two issues were listed here and must not close, re-audited 2026-09-02 against the branch:

• test(framework): a diagnostic that proves the chain fires, and a period that has met a hundred sessions #694 — the diagnostic it asks for exists and is verified live; the hundred-session load evidence and the timed walk it also asks for do not.
• fix(cli): a project the CLI set up loads none of its plugins headless, and says nothing #703 — this pull request's own body says the "distinguishable from one that needed none" criterion has no implementation. It cannot close against itself.

#676 was in that block until today and no longer belongs there. Its last box asks that docs/ARCHITECTURE.md record the second installation mode, as an exception to "hooks are declared in hooks.json" — and c10c2222, on this branch, does exactly that: the OpenCode row names hooks/opencode-plugin.js, the three events it maps, and says in as many words that the column it sits in is about the declarative axis alone.

#649 closes as a rescope, not as written. Its scope asks for a metadata.json carrying unit_id, an upward backlog link, branch, pull_request, and a steps[] journal. Only the upward link is built. The rest was argued redundant against this branch and not merely skipped: the journal already timestamps step_start and file_written, so which step produced which file is derivable by the same interval mechanism task attribution uses, and a second file would be a second source of truth that can disagree with the first; the task folder path is already an identity; branch and pull_request are derivable from git and the forge. Anyone reading #649's scope against the branch will find four of its five fields absent, which is the intended outcome, not a gap.

Two more closed separately, because the branch removed their subject rather than satisfying them: #653 (its outcome arrived by the local-read route; the export route it describes was deleted) and #655 (it redacts an upload path that no longer exists — its outbound clause moved to #662).

✅ I certify

• I DO CERTIFY I READ EACH LINE OF THE PULL REQUEST BECAUSE I AM A SOFTWARE ENGINEER, NOT A AI PUPPY.

[blafourcade]

What to read, and why the file count lies

GitHub says 1801 files. 142 of them are this work.

The branch was cut on 16 July at chore: release main (#447). Since then main moved 115 commits and this branch moved 165, and in that window main received the CLI migration while this branch carries its own copy of it (10bdd605 fix(cli): migrate aidd-cli into framework as cli/). The three-dot diff a pull request shows is measured from the merge base, so it counts that migration twice — every file under cli/src, cli/tests, kanban and the other plugins is in the total for that reason and not because anything here touched them.

The eleven commits that are this work, from a0beac5b to 1015566f:

142 files changed, 10,518 insertions(+), 418 deletions(-)

  32  plugins/aidd-telemetry     the hooks, the three skills
  31  cli/tests                  and 29 cli/src
  28  aidd_docs/tasks            the plans, phases and measurements
  13  scripts/__tests__          the plugin's own suite
   3  docs                       what each tool can and cannot measure

Compare view for exactly those: 0509345...1015566

The base needs a decision before this leaves draft

Three ways out, none of which I will take unasked because they all rewrite or reshape someone else's branch:

1. Rebase onto current main. The duplicated migration should collapse and the diff would show its true size. It rewrites 165 commits of history.
2. Merge main in. No history rewritten, and it fixes the diff for the same reason — the merge base becomes main's tip. I dry-ran it: 90 conflicts, 63 of them add/add on the duplicated migration, 38 under cli/src and 20 under cli/tests. Resolvable, and not a thing to do quickly.
3. Leave it and review by range, using the link above.

Everything else is ready

Eleven commits, each passing the pre-commit hook without modification. The pre-push run: 252 test files, 2680 tests. Plugin scripts 365, CLI unit 1931, integration 577, e2e 178, tsc clean, biome clean, no broken markdown links.

Every per-tool claim in the body rests on a session that was actually run, and the negative results are in aidd_docs/tasks/2026_08/2026_08_22_telemetry-every-tool/measurements.md in full — including the two probes that disagreed about OpenCode and the measurement that reconciled them.

[blafourcade]

Retargeted to next, which is where it should have gone

CONTRIBUTING.md:53 says "Branch off next, target next". This was opened against main, which is the branch promote.yml rebase-merges next into — not one a feature branch belongs on. Fixed.

It also shrinks the count, because the merge base moves from 16 July to 13 August:

against main    1801 files   +164,300
against next     368 files    +45,625
this work        142 files    +10,518

Still not 142, and the reason is unchanged: the branch carries its own copy of the CLI migration that both main and next have since received, so the diff counts it twice. The comparison range in the previous comment is still the thing to read.

The base decision stands, and it is now smaller

Merging next in was dry-run against main and produced 90 conflicts. That number will differ against next and I have not re-measured it. The three options are the same — rebase, merge, or review by range — and none is mine to take on someone else's 165 commits.

Not merged, deliberately

The eight Closes fire on merge, so the backlog closes when this does. I have not merged it and will not: this is a shared repository with a release process (promote.yml rebase-merges next into main, back-merge.yml returns the other way), a draft pull request nobody has reviewed, and 165 commits of which 154 are not mine. Landing that on a shared branch is a call for whoever owns the release train, not for the author of the last eleven commits.