Skip to content

fix(cli, llm): Correct the token estimate, stop gating summaries - #1146

Open
JeanMertz wants to merge 1 commit into
mainfrom
token-counting
Open

fix(cli, llm): Correct the token estimate, stop gating summaries#1146
JeanMertz wants to merge 1 commit into
mainfrom
token-counting

Conversation

@JeanMertz

Copy link
Copy Markdown
Collaborator

Summarizing a range that does not fit the model's context window now reports the provider's own rejection, carrying the request's real token count. Running jp conversation compact --turn=2..50 --summary on a large conversation passed a local character-based check and then failed with a bare stream error; it now reports a summarization failure naming the model, the actual token count against the window, and the two ways to get under it.

The pre-flight check is removed rather than made accurate. A token count request ships the same payload as the request it would guard, so it saves nothing when the range is too large and costs an extra upload every time the range fits. The provider's message is also more precise than anything measured locally, and the error it raises was already classified as ContextWindowExceeded.

CHARS_PER_TOKEN drops from 3 to 2. A 4,220,150-byte Anthropic request body counted 1,317,976 input tokens, which is roughly 1.9-2.0 chars per token once JSON framing is backed out, so the old ratio overstated the available window by about half. This tightens the truncation budget for title generation and tool inquiries too, since they share the same estimator.

One detail is no longer reported: the failure message does not name the turn range, which the caller supplied on the command line and the provider's token counts describe more usefully.

Summarizing a range that does not fit the model's context window now
reports the provider's own rejection, carrying the request's real token
count. Running `jp conversation compact --turn=2..50 --summary` on a
large conversation passed a local character-based check and then failed
with a bare stream error; it now reports a summarization failure naming
the model, the actual token count against the window, and the two ways
to get under it.

The pre-flight check is removed rather than made accurate. A token count
request ships the same payload as the request it would guard, so it
saves nothing when the range is too large and costs an extra upload
every time the range fits. The provider's message is also more precise
than anything measured locally, and the error it raises was already
classified as `ContextWindowExceeded`.

`CHARS_PER_TOKEN` drops from 3 to 2. A 4,220,150-byte Anthropic request
body counted 1,317,976 input tokens, which is roughly 1.9-2.0 chars per
token once JSON framing is backed out, so the old ratio overstated the
available window by about half. This tightens the truncation budget for
title generation and tool inquiries too, since they share the same
estimator.

One detail is no longer reported: the failure message does not name the
turn range, which the caller supplied on the command line and the
provider's token counts describe more usefully.

Signed-off-by: Jean Mertz <git@jeanmertz.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant