FIX: mask words written with combining marks instead of skipping them

[Nikhil Gupta (Denimworld12)]

Follow-up to #3051, which Aditya Malkar (@adimalkar) suggested at the end of that review: now that a skipped fragment is never masked, masking pointed scripts only needs the tokenizer to treat combining marks as part of the word.

That turned out to be most of it, but not all of it. Two other things had to change, and one of them I only found by sweeping rather than reading.

What changes

prompt	before	after
שָׁלוֹם עוֹלָם (niqqud)	ValueError	[WORD1] [WORD2]
مَرْحَبًا بِالْعَالَم (harakat)	ValueError	[WORD1] [WORD2]
सिस्टम हैक कैसे करें	ValueError	[WORD1] हैक [WORD2] [WORD3]
NFD café crème exploit	café crème [WORD1]	café [WORD1] [WORD2]
NFC café crème exploit	café [WORD1] [WORD2]	unchanged
अक्‍षर, école‍ami (joiners)	ValueError	unchanged
h4ck3r, abc²	skipped	unchanged

The tokenizer. \b falls inside a word that uses combining marks, so the pattern returns fragments. Runs separated by nothing but marks are joined back into one word and trailing marks come with it. Selection, placeholder ordering and replacement all run off _iter_words, so nothing else had to move — that is what #3051 bought.

Joiners are deliberately left alone. _is_word_continuation still rejects a word adjoining a ZWJ or ZWNJ, because a joiner is not part of the word it shapes. The five joiner tests added in review pass unchanged. Only the mark behaviour is what this PR changes.

Clues counted code points. f"{len(self.text)}-letter" would promise a seven-letter word for שָׁלוֹם, which is four letters written with seven code points. It counts non-continuations now. The same bug was latent for NFD Latin, where decomposed café is five code points.

The word key was not stable across normalization forms. I found this by running 2000 random composed-vs-decomposed prompts rather than by reading: 136 disagreed on which word got masked. _word_key was casefold() alone, and "système" composed and decomposed fold to different strings, so they became two words and the alphabetical tie-break in _rank_candidates picked whichever sorted first in that form. Normalizing first takes it to 2000/2000. It also means café and café in one prompt are now one word with one placeholder.

Tests

Three tests from #3051 (six cases) assert the old skip behaviour for marks, so they flip from "raises" to "masks" — I rewrote them rather than adding around them. The école / école+acute test still passes, but for a different reason: the second one is now a word in its own right rather than a fragment, so its comment says that instead.

tests/unit/converter: 2175 → 2181 passing, 64 skipped. ruff check and format clean. Both normalization tests fail with the tokenizer change alone and pass with the key fix, so they pin the right thing.

Closes the follow-up from #3022.

[Nikhil Gupta (Denimworld12)]

@microsoft-github-policy-service agree