fix(fast): apply n-ary merges and match the reference tie-break exactly

[AmitMY]

Closes #47. Both halves, one goal: fast now produces byte-identical output to the reference on every benchmark case.

Fix 1 — n>2 merges never applied (seq_try_merge)

The (!has_complex_child && only_minimal) short-circuit assumed minimal merges are always pairs, so any merge of size > 2 silently failed to apply through try_merge — BNE's default path picked a 4-node merge, applied nothing, and re-picked the same merge every step. The guard is now just n < m, matching seq_merge's semantics.

Fix 2 — tie-break: bytes → traversal order

The reference (pinned to HuggingFace by test_bpe_matches_huggingface_merges) takes the first tied candidate in traversal order; fast took the largest by assembled bytes. Designed so the common path pays nothing:

• best_and_tie() finds the max in the same single pass as before and returns immediately when the max is unique — zero extra allocation, which is why BPE/Boundless times are unchanged.
• On a tie, an early-exit walker (first_merge_in) mirrors emit_merges order and short-circuits at the first tied candidate; the delta path uses the existing inverted index to jump straight to the earliest entry containing a tied candidate. (A naive full re-emit per tie regressed Boundless 0.174s → 0.250s; the early exit restored 0.174s.)
• Entry grouping now preserves first-occurrence order (was HashMap order), so scan order equals traversal order on every path (train_unconn, delta, single, streaming).
• The dead bytes-ranking machinery (HF-rank table etc.) is removed.

Output & performance (Zipf bench, wikitext-agnostic digests)

Case	fast before	fast after	reference	time before → after
BPE	f4bb575b8e	05d1281ec2	05d1281ec2 ✓	0.033s → 0.038s
BNE n=4	59ac6c0fa0 (degenerate)	e52c6db4bb	e52c6db4bb ✓	0.036s → 0.066s*
Boundless	5089559be4	c045c0e773	c045c0e773 ✓	0.174s → 0.173s

* BNE's old time was the bug: it never applied its merges. 0.066s is the honest cost of correct n-ary training (reference: 0.195s).

Tests

• New shared test tests/tokenizers/test_tie_break.py: a corpus with a verified 7-way first-merge score tie + n=3 merges, asserting hardcoded reference merge lists — pins the contract for both implementations.
• Both suites: 141 passed, 2 skipped each. Clippy: no new warnings (one pre-existing warning removed with the dead code).

With this, the benchmark digest column (#51) converges across implementations — any future divergence is a regression, visible at a glance.

🤖 Generated with Claude Code