Repository navigation
Conversation
Add native MLX inference for GigaAM v3 on Apple Silicon: - Full Conformer encoder (16 layers, 768d) with RoPE attention - CTC and RNNT heads with greedy decoding - PyTorch → MLX weight conversion (safetensors + config.json) - Streaming transcription (growing buffer, live mic + file) - CLI tools: gigaam-cli, gigaam-stream, gigaam-transcribe - Python API: load_model, transcribe, stream_generate, stream_live Performance on Apple M4: - CTC: 139x realtime (11s audio in 81ms), fp16 421 MB - RNNT: 48x realtime (11s in 230ms), ~9% lower WER than CTC New files in mlx_convert/: gigaam_mlx.py — MLX model + inference + streaming convert_gigaam_to_mlx.py — PyTorch → MLX conversion gigaam-cli — single-file transcription CLI gigaam-stream — real-time streaming CLI gigaam-transcribe — shell wrapper README.md — documentation, API, benchmarks
|
any chances for this PR to be merged? |
- RoPE base: upstream builds RotaryPositionalEmbedding(d_model // n_heads, pos_emb_max_len), whose second positional parameter is `base`, so the base is 5000. The port used 10000: encoder outputs were off by up to ~1.8 while transcripts mostly still matched. Now max |Δ| ≤ 9e-5 vs PyTorch. - E2E decoding: E2E CTC / RNNT output SentencePiece ids; the runtime joined raw pieces. decode_sentencepiece reproduces SentencePieceProcessor.decode (control ids, <unk>, <0xNN> byte fallback, dummy prefix) without the sentencepiece package. - Converter takes the vocabulary from model.decoding.tokenizer: fixes E2E CTC (previously got the 33-char vocabulary) and writes tokenizer.model from the loaded model instead of guessing a cache path. - Keep mel_filterbank / stft_window in fp32 in the fp16 file. - load_model upcasts weights to fp32 by default (dtype=None keeps stored): activations are fp32 anyway and fp16 weights cost a cast per op, ~20%. - gigaam-stream caps MLX's buffer cache at 1 GB: every step has a new input length and the cache otherwise grows towards most of unified memory.
eval/ runs upstream PyTorch and the MLX port on the 200-clip Golos Crowd sample of garrrikkotua/gigaam-vs-whisper-russian (downloaded from the HF mirror and verified by SHA-256; audio not redistributed) and reports: - parity: identical transcripts 200/200 for v3_ctc, v3_rnnt, v3_e2e_ctc, v3_e2e_rnnt; encoder max |Δ| ≤ 9e-5; CTC frame labels 100% identical; - WER/CER with a documented normalization (v3_rnnt: 15/1001, identical to the archived PyTorch result of that benchmark); - speed vs PyTorch MPS / CPU, one subprocess per run, MLX cache capped. README: replace the unmeasured benchmark table with these results.
misteral
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Native MLX inference for GigaAM v3 on Apple Silicon: v3_ctc, v3_rnnt, v3_e2e_ctc, v3_e2e_rnnt.
Transcripts are identical to the PyTorch model on 200/200 evaluation clips for all four models.
Supersedes #61 (rebased on main, dev artifacts removed).
What's included
mlx_convert/gigaam_mlx.pymlx_convert/convert_gigaam_to_mlx.pymlx_convert/gigaam-cli,gigaam-stream,gigaam-transcribemlx_convert/eval/mlx_convert/README.mdEvaluation
Apple M4 Pro (24 GB), mlx 0.32.2, torch 2.14.0; 200 Golos Crowd test clips (823.85 s) — the sample of
garrrikkotua/gigaam-vs-whisper-russian, verified by SHA-256.
Full tables:
mlx_convert/eval/results/RESULTS.md.Parity vs PyTorch (CPU, fp32): identical transcripts 200/200 for all four models; encoder output max |Δ| ≤ 9e-5
per clip; CTC frame labels 100% identical. v3_rnnt: 15 errors / 1001 words, the same hypotheses as the archived
PyTorch run of that benchmark.
WER (E2E output normalized; last row excludes clips where E2E output contains digits/Latin — formatting, not recognition):
Speed, × realtime (short = 200 clips one by one, median 3.9 s / long = ≈20 s segments):
For CTC models MLX is on par with PyTorch MPS; for RNNT models MLX is 2.3–2.5× faster. The earlier figures in this
PR (139× / 48×) were not produced by a reproducible benchmark and are replaced by the above.
Changes since the first version of this PR
RotaryPositionalEmbedding(d_model // n_heads, pos_emb_max_len)passespos_emb_max_lenasbase; the port used 10000. Transcripts still matched on most clips (195–199/200 forCTC/RNNT, WER unchanged), but encoder outputs differed by up to ~1.8 — found by the tensor-level parity check.
sentencepiecepackage, verified againstSentencePieceProcessor.decode(eval/test_decode.py); converter takes the vocabulary from the model's tokenizer(v3_e2e_ctc previously got the 33-character vocabulary).
load_modelupcasts the fp16 weights once at load(activations are fp32 anyway; ~20% faster),
dtype=Nonekeeps them.gigaam-streamcaps MLX's buffer cache (1 GB): with a new input length every step it otherwise grows towardsmost of unified memory.
Architecture notes
[out, in, K]→[out, K, in]; LSTMbias_ih + bias_hh→ one bias.