Repository navigation
perf: memoize get_merges (TRADE_MEMORY_FOR_SPEED, default True) - #30
Merged
Merged
Conversation
get_merges is a pure function of an immutable node, and merge returns self for unchanged subtrees, so the same node objects recur across merges and a full re-walk becomes a cache hit. NodesSequence now memoizes its candidate list when GraphSettings.TRADE_MEMORY_FOR_SPEED is set. Default True: ~35-40% faster across all tokenizers at roughly 2x peak memory. Set False to keep the lower-memory behavior. Output is identical either way; a new test asserts the memoized path matches the unmemoized one. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
get_mergesis a pure function of an immutable node, andmergereturnsselffor unchanged subtrees (PR #28), so the same node objects recur across merges — a full re-walk every step can instead be a cache hit.NodesSequencenow memoizes its candidate list, gated by a newGraphSettings.TRADE_MEMORY_FOR_SPEEDflag:Default True — favors speed. Set it to
Falseto keep the previous lower-memory behavior. Output is identical either way (a new test trains with the flag off and on and asserts the merge sequences match). All 137 tests pass.Measured (50 texts / 200 merges):
~35–40% faster (grows toward ~1.65× at larger scale) for roughly 2× peak memory. Notably this also speeds up the connected tokenizers (Boundless/SuperBPE).
🤖 Generated with Claude Code