Fix weights_only=True load failure for bundled voice presets - #440
Open
Ammar Hashmi (ammar-iitm) wants to merge 2 commits into
Open
Ammar Hashmi (ammar-iitm) wants to merge 2 commits into
Ammar Hashmi (ammar-iitm) wants to merge 2 commits into
Conversation
demo/web/app.py crashed on startup with "Weights only load failed" on recent PyTorch (not ROCm-specific) because the bundled voice preset .pt files pickled BaseModelOutputWithPast/DynamicCache objects. Even with safe_globals allowing those classes, torch's weights-only unpickler can't reconstruct them: it only supports the SETITEMS restore op for plain dict/OrderedDict/Counter, and these are subclasses that don't qualify (confirmed root cause, see microsoft#392). Re-save the 25 bundled presets so each stores only plain dict/list/int/Tensor values (last_hidden_state tensor + past KV cache tensors/seen_tokens), which the default weights-only unpickler already supports. _ensure_voice_cached now loads that plain structure with weights_only=True (no safe_globals needed) and reconstructs the BaseModelOutputWithPast/DynamicCache objects in code. Verified in an isolated venv: every converted preset round-trips bit-for-bit identical to the original tensors, loads under weights_only=True, and survives the copy.deepcopy() used downstream in stream(). Fixes microsoft#392 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016fM2LafALmBK16KnFWjsp7
The conversion script wrote replacement files via a temp file, which picked up the default umask instead of the original mode. No content change. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016fM2LafALmBK16KnFWjsp7
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
demo/web/app.pycrashes on startup on recent PyTorch (not ROCm-specific — reproduced on CPU with torch 2.14 / transformers 5.16) with:.ptfiles pickleBaseModelOutputWithPast/DynamicCacheobjects. Even withsafe_globalsallowing those classes, torch's weights-only unpickler only supports theSETITEMSrestore path for plaindict/OrderedDict/Counter, not subclasses like these, so the load fails regardless ofsafe_globals.weights_only=Falsestopgap.Changes
demo/voices/streaming_model/*.ptso each stores only plaindict/list/int/Tensorvalues (last_hidden_statetensor, pluspast_key_valuesas{seen_tokens, key_cache: [...], value_cache: [...]}) instead of pickling theBaseModelOutputWithPast/DynamicCacheobjects directly.StreamingTTSService._ensure_voice_cachedindemo/web/app.pynow loads that plain structure withtorch.load(..., weights_only=True)(nosafe_globalsneeded) and reconstructsBaseModelOutputWithPast/DynamicCachein code before caching.Verification
Done in an isolated venv (torch 2.14.0 cpu, transformers 5.16.1) since I don't have a GPU here:
.ptfiles with the exactsafe_globals+weights_only=Truecall fromapp.py.weights_only=False(trusted, since these are the repo's own bundled assets), converted to the new plain format, saved, then reloaded withweights_only=Trueand asserted the reconstructedlast_hidden_state,seen_tokens,key_cache, andvalue_cachetensors are bit-for-bit identical (torch.equal) to the originals, before overwriting each file._ensure_voice_cachedlogic verbatim against all 25 converted files — all load cleanly underweights_only=Truewith nosafe_globals.copy.deepcopy()on the reconstructedprefilled_outputs(used downstream instream()/_run_generation) still works.python3 -m py_compile demo/web/app.pypasses.Not verified: full end-to-end streaming generation on GPU (no GPU available in this environment) — would appreciate a maintainer or the original reporter confirming the demo now starts cleanly end-to-end.
🤖 Generated with Claude Code
https://claude.ai/code/session_016fM2LafALmBK16KnFWjsp7