Repository navigation
v1.6.21: accuracy release (token counts, pricing, honest benchmark wording) - #53
Merged
Merged
Conversation
modelLabel() returned a hardcoded "opus 4.8"/"sonnet 4.6"/"haiku 4.5" for any opus/sonnet/haiku id — the one place in the codebase a model version was baked into display code (would rot on the next model, and could disagree with the real id). Now derives "<family> X.Y" from the id (e.g. claude-opus-4-8 -> "opus 4.8") with an optional-minor fallback. Current output is byte-identical; test fixtures updated to real ids (claude-opus-4-8 etc.) so the snapshot reflects reality. Note: why/stats/impact already read the model per-session from the transcript (session.primaryModel) — verified against a real transcript recording claude-opus-4-7 in 1,061 messages — so those were never hardcoded. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…usion Follow-up to the derivation fix: rather than show "opus 4.8", estimate now labels rows by TIER (opus / sonnet / haiku) with no version number at all. Rationale: estimate is a forward cost comparator and prices per tier (every opus minor version costs the same), so the tier is the meaningful unit. And without a version, the estimate can NEVER show a model version that disagrees with what a user's session actually used (that exact version is reported by why/stats/impact, read from the transcript) and it can never rot. Recommendation line drops the now-redundant model prefix (the reason text already names the tier, e.g. "sonnet is the sweet spot"). Snapshot + assertions updated. Full suite 1,389 green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`sipcode init` imports getRegisteredMcpToolCount from mcp/server.js.
The server called main() and registered a `process.stdin.on("end")`
handler at module load, so the import started a real server. Under
non-TTY stdin (CI, pipes, `init --yes`) that handler saw an immediate
EOF and called process.exit(0), killing init mid-runManifest — init
printed its header, exited 0, and created no files.
Guard all process-level handlers + main() behind an _isMainModule
check so startup happens only when run as the sipcode-mcp binary.
Importing the module is now side-effect-free.
Adds an e2e regression test: importing the module with EOF stdin must
return 15 tools without booting the server or exiting the host.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude Code writes one JSONL line per content block of a response and repeats the request's usage on every line. The parser summed every line, so all token and cost totals (stats, today, why, forecast, trend, impact, receipt, estimate anchors) counted each request 2-3x. Measured on real sessions: 2.06x and 2.35x. Lines are now merged by message.id + requestId (max per usage field, tool calls collected from every line). Duplicate-read and idle-context costs charged each call the request's whole cached context. They now use the tokens the call's tool_result actually added (chars / 4), so Claude Code's native "file unchanged" stub costs ~nothing. Also reads the nested 5m/1h cache_creation split when the top-level field is 0 (Claude Code < 2.1.152). New regression tests fail on the old parser and pass on the new one. The read-heavy fixture now returns file-sized results instead of the literal "file contents". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…cache writes - Opus 4.5-4.8 were priced at Opus 4's $15/$75; official is $5/$25 (3x overstatement). Haiku 4.5 was $0.80/$4; official is $1/$5. - 5.x models (Opus 5/5.5, Sonnet 5/5.5, Fable 5/5.1, Haiku 5.5) had no row, so prefix matching found nothing and they cost $0. New 2026-10-08 table from platform.claude.com/docs/en/about-claude/pricing. - Model matching now takes the longest matching key (dated ids such as claude-haiku-4-5-20251001 no longer fall back to a generic row). - Older dated tables fill missing models from the newest table. - 1-hour cache writes (2x input; Claude Code subscriptions use them for the main conversation) are priced separately from 5-minute writes (1.25x). Haiku 5.5's >100K-prompt tier is applied per request. - estimate predicts against Opus 5.5 / Sonnet 5.5 / Haiku 4.5 and matches models by tier; anchors weight 1h writes at 2x. Note: Anthropic's page lists Sonnet 5.5 cache reads as $0.20 in the model table and $0.10 in the caching section; the model table is used. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A venv's Lib/, include/ and Scripts/ were scanned into the manifest. Environments are now detected by marker, whatever the folder is named: pyvenv.cfg (venv/virtualenv) or conda-meta/ (conda). A nested env is skipped entirely; a venv created in the project root itself skips only its own Lib/, include/, Scripts/, bin/, lib/, lib64/ and share/ folders. site-packages is also a hard skip. Fixes #22 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 20 transcript pairs were generated by benchmark/_gen/build-corpus.mjs, not captured from live Claude Code runs, but METHODOLOGY.md called them "captured" sessions. Every surface that cites 62.6% (README, site hero, demo, layers, FAQ, compare page, llms.txt, llms-full.txt, benchmark skill, impact hint) now says the corpus is synthetic and that real-world savings depend on the workload. The benchmark dollar total is now $22.48, not $67.43: the corpus model claude-opus-4-7 was priced at Opus 4's $15/$75 instead of $5/$25. The llms.txt claim "at current Claude Sonnet pricing" was also wrong. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Version bump, CHANGELOG, server.json, site and llms.txt version and test counts (1,404). No tag yet: tagging triggers the npm publish. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An accuracy release. Several of Sipcode's numbers were wrong in ways that matter; this fixes them and says so in the CHANGELOG.
Fixed
message.id+requestId. Real machine: 29.1B tokens reported vs 13.1B actual.init --yesin non-TTY contexts (scripts, CI) exited early because importing the MCP module started the server.pyvenv.cfg/conda-meta/.estimate: tier-only model labels; predicts against Opus 5.5 / Sonnet 5.5 / Haiku 4.5.Changed
Internal
ci.yml: typecheck, build, unit + e2e tests on every PR and push to main (Ubuntu + Windows). Closes CI: test suite does not run on PRs or pushes to main #33, closes CI: test suite does not run on PRs or pushes to main #38.Fixes #22
🤖 Generated with Claude Code