fix(cli, llm): Correct the token estimate, stop gating summaries - #1146
Open
JeanMertz wants to merge 1 commit into
Open
fix(cli, llm): Correct the token estimate, stop gating summaries#1146JeanMertz wants to merge 1 commit into
JeanMertz wants to merge 1 commit into
Conversation
Summarizing a range that does not fit the model's context window now reports the provider's own rejection, carrying the request's real token count. Running `jp conversation compact --turn=2..50 --summary` on a large conversation passed a local character-based check and then failed with a bare stream error; it now reports a summarization failure naming the model, the actual token count against the window, and the two ways to get under it. The pre-flight check is removed rather than made accurate. A token count request ships the same payload as the request it would guard, so it saves nothing when the range is too large and costs an extra upload every time the range fits. The provider's message is also more precise than anything measured locally, and the error it raises was already classified as `ContextWindowExceeded`. `CHARS_PER_TOKEN` drops from 3 to 2. A 4,220,150-byte Anthropic request body counted 1,317,976 input tokens, which is roughly 1.9-2.0 chars per token once JSON framing is backed out, so the old ratio overstated the available window by about half. This tightens the truncation budget for title generation and tool inquiries too, since they share the same estimator. One detail is no longer reported: the failure message does not name the turn range, which the caller supplied on the command line and the provider's token counts describe more usefully. Signed-off-by: Jean Mertz <git@jeanmertz.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summarizing a range that does not fit the model's context window now reports the provider's own rejection, carrying the request's real token count. Running
jp conversation compact --turn=2..50 --summaryon a large conversation passed a local character-based check and then failed with a bare stream error; it now reports a summarization failure naming the model, the actual token count against the window, and the two ways to get under it.The pre-flight check is removed rather than made accurate. A token count request ships the same payload as the request it would guard, so it saves nothing when the range is too large and costs an extra upload every time the range fits. The provider's message is also more precise than anything measured locally, and the error it raises was already classified as
ContextWindowExceeded.CHARS_PER_TOKENdrops from 3 to 2. A 4,220,150-byte Anthropic request body counted 1,317,976 input tokens, which is roughly 1.9-2.0 chars per token once JSON framing is backed out, so the old ratio overstated the available window by about half. This tightens the truncation budget for title generation and tool inquiries too, since they share the same estimator.One detail is no longer reported: the failure message does not name the turn range, which the caller supplied on the command line and the provider's token counts describe more usefully.