Repository navigation
perf: pick best merge over counts.items() - #34
Merged
Merged
Conversation
max(counts, key=lambda k: (len(k)-1)*counts[k]) did a dict lookup counts[k] for every candidate. Iterating counts.items() hands the count straight to the key function, skipping the re-lookup. Same iteration order, so the merge chosen (and its tie-break) is identical. ~3% faster for BPE/Boundless, ~9% for BNE (more candidates -> more saved lookups). Output identical; 137 tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
AmitMY
commented
Jun 26, 2026
| break | ||
|
|
||
| nodes = max(counts, key=lambda k: (len(k) - 1) * counts[k]) | ||
| nodes = max(counts.items(), key=lambda kv: (len(kv[0]) - 1) * kv[1])[0] |
Contributor
Author
There was a problem hiding this comment.
is it possible to have them named in the lambda, like "lambda (key, value): ...."?
Addresses review: lambda tuple-unpacking (lambda (k, v): ...) was removed in Python 3 (PEP 3113), so a small named function gives the readable (nodes, count) unpacking instead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
Author
|
Good idea, but Python 3 removed tuple-parameter unpacking (PEP 3113), so |
Contributor
Author
|
Verified after the refactor: 137 tests pass and the merge-sequence digests are identical to baseline, so behavior is unchanged. Speed is the same as the inline-lambda version (a named function call has identical overhead) — vs main: BPE 0.846→0.818s (−3.3%), BNE 1.452→1.312s (−9.6%), Boundless 0.982→0.933s (−5.0%); memory unchanged. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Choosing the best merge did a dict lookup per candidate:
Iterating
counts.items()hands the count straight to the key function:Same iteration order (insertion order either way), so the chosen merge and its tie-breaking are identical — the exact-merge parity tests cover this.
Output identical, 137 tests pass. Measured back-to-back (50 texts / 200 merges; time = best of 3):
BNE gains most — it has the most candidates (up to 4-grams), so the most saved lookups. Memory unchanged.
🤖 Generated with Claude Code