Cloning an Italian voice with S2-Pro and LoRA finetuning — 7M trainable params, 55h dataset, fast-AR only

[Enucatl]

I fine-tuned S2-Pro to clone the voice of a specific Italian public figure using LoRA, and wrote up the full process.

Audio samples (step 1100 and 2100, same prompt):
"Fermi tutti! Questo programma può contenere riferimenti espliciti e si rivolge ad un pubblico adulto. E non è adatto... ai minori!"

sample_step_1100.mp4

sample_step_2100.mp4

Key findings:

• Fast-AR-only LoRA (r=32, alpha=16) is sufficient and safe — 7.1M of 4.6B params, slow transformer fully preserved
• Touching the slow transformer causes rapid degradation, especially at higher alpha and learning rate
• Convergence is good already by ~1000 stepsm which can be done in about 1h on a single RTX 5090.

Detailed write-up: https://github.com/Enucatl/fish-speech/blob/barbero/lora-finetune.md

Five upstream PRs came out of this work: #1225 (CUDA 12.9 / Blackwell support), #1227 (progress bar fix for gradient accumulation), #1229 (protobuf conflict fix), #1231 (target_modules with fast_ prefix support), and #1233 (AudioSampleCallback for in-training audio monitoring).

[Mixomo]

Great report! Just what I was looking for!

I’m planning to fine-tune a LoRa model with a Spanish voice (Argentine accent).

Does it absolutely have to be several hours of audio, or could it be just a few minutes? (20 / 30 / 60 minutes)

Do you think it can be trained on a 3090 with 24 GB of VRAM? Or do the model’s size and parameters require at least 32 GB of VRAM?

Thanks!

[Mixomo]

Honestly, I haven't tried it, as it's a feature that doesn't interest me for my use case, but LoRa should retain the tag feature, since it's combined on top of the base model.

[ryuhda]

Honestly, I haven't tried it, as it's a feature that doesn't interest me for my use case, but LoRa should retain the tag feature, since it's combined on top of the base model.

Thanks for the GUI link, downloading it now! A few quick questions:

1. How many samples did you use for your tests, and roughly how much total audio duration?

2. For gathering samples, do you think quality beats quantity, or the other way around?

3. Does your GUI apply LoRA to Fast AR only (like Enucatl's approach), or to all modules including Slow AR? Asking because Fast AR only should preserve the base model's emotion tag behavior better.

4. Did you actually complete a full LoRA training run and compare the result to zero-shot voice cloning on the same reference? Trying to figure out if LoRA meaningfully improves over zero-shot for a small-to-medium dataset. Even a rough impression would help!

Thanks!

[Enucatl]

1. in my case it's about 8k files totalling 55h of video
2. quality beats quantity
3. I have a comparison to the zero-shot in the report. The improvements are meaningful, even if the zero-shot is quite good already.

[ryuhda]

1. in my case it's about 8k files totalling 55h of video
2. quality beats quantity
3. I have a comparison to the zero-shot in the report. The improvements are meaningful, even if the zero-shot is quite good already.

OMG thank you so much. you are my savior hahaha

[ryuhda]

1. in my case it's about 8k files totalling 55h of video
2. quality beats quantity
3. I have a comparison to the zero-shot in the report. The improvements are meaningful, even if the zero-shot is quite good already.

Thanks again for the detailed writeup. One follow-up on the emotion side:
Since Italian sits in Fish's Tier 3 "global coverage" bucket, the base model technically knows it but the coverage is thin. After your 55h Fast-AR-only LoRA run, did you ever test whether the emotion tags (crying, sobbing, shouting, sad, etc.) still fire on Italian prompts? In theory Fast-AR-only should leave the slow transformer — and therefore tag behavior — untouched, but I'm curious whether you actually observed that in practice, or if the heavy Fast-AR specialization ended up dampening tag responsiveness even on a thin-tier language.
Any anecdotal impression would help a lot. Thanks!

[willbot54]

This is my first time posting on Github, lmao. I've been working on this for hours today. Has anyone had any luck using Linux and AMD maybe ROCm? It's so finicky. I'm trying to clone a character voice and Lora is not working as it looks like this project and everything else needs NVIDIA CUDA :( I wish they could update the source code so that people with AMG GPUS and Linux could run this and make their own voices for offline use without cloud.

[Mixomo]

@ryuhda The GUI code follows the same method as Matteo. Since the base model is so mature, I haven't seen significant improvements training small datasets, which is why I marked the training section as experimental.
I suppose the results are more noticeable with datasets of many hours. And yes, it only trains the fast module.

@willbot54 Yes, Unfortunately, the entire foundation of AI runs on CUDA. There's something for AMD called Zluda, which translates CUDA calls into a language AMD can understand, but I'm not familiar with that.

[Enucatl]

I tested it and it works similarly to the base model

…

------------- Matteo Sent from [Proton Mail](https://proton.me/mail/home) for Android.

-------- Original Message --------

On Wednesday, 04/15/26 at 03:50 ryuhda ***@***.***> wrote: > - in my case it's about 8k files totalling 55h of video > - quality beats quantity > - I have a comparison to the zero-shot in the report. The improvements are meaningful, even if the zero-shot is quite good already. Thanks again for the detailed writeup. One follow-up on the emotion side: Since Italian sits in Fish's Tier 3 "global coverage" bucket, the base model technically knows it but the coverage is thin. After your 55h Fast-AR-only LoRA run, did you ever test whether the emotion tags (crying, sobbing, shouting, sad, etc.) still fire on Italian prompts? In theory Fast-AR-only should leave the slow transformer — and therefore tag behavior — untouched, but I'm curious whether you actually observed that in practice, or if the heavy Fast-AR specialization ended up dampening tag responsiveness even on a thin-tier language. Any anecdotal impression would help a lot. Thanks! — Reply to this email directly, [view it on GitHub](#1234 (reply in thread)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AAK3SVNVOYVQ2ZYWGYRPCST4V3TEPAVCNFSM6AAAAACW2ST75GVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTMNJWGM4DGOI). You are receiving this because you authored the thread.Message ID: ***@***.***>

[koushikkonwar]

Hi,

I am currently full fine-tuning S2-Pro on my internal multi-speaker dataset using speaker tags. I’ve observed that the generated audio starts off well, but gradually degrades toward the end of the utterance after fine-tuning.

If anyone has faced a similar issue and was able to resolve it, I’d really appreciate hearing your insights or suggestions

[Enucatl]

I mention this in the full report, and this is why I think a fast-only lora is preferrable.

You could go for a full fine-tuning but drastically reduce learning rate.