Step-by-Step Guide to Run S2-Pro on 2x 16gb cards

[Jan21021]

Hey All!

First of a all, I'm a complete LLM TTS noob, so a lot of this stuff was a first for me.

I had the following Problem: I have two consumer RTX cards, a 5070 Ti and a 4080 Super. Both with 16gb ram. Since the S2-Pro Models needs 24 GB of ram, I was not able to run the s2-pro model.
I ran it in CPU Mode and really, really liked the quality and the tag systems. that's wy I even bothered to get a working solution.
So i did waht a noob does and thats trying to get it to work with a LLM. Gemini sucked hard, so naturally i tried claude. And yeah, he got a solution.
Can't belive it by myself but it works!

I don't make a Pull Request since I think this is somewhat a special problem that one has two 16gb Cards.
So if anybody has the same problem, you can try the following steps.

I don't want to just copy paste a AI Slop output so I added some stuff so that another noob like me can understand it.
This guide targets beginner like me, if you're a pro, just skip to the part where we do the patches so that it runs on two GPUs, if that's what you want to do.

===========
All Commands show here in this guide are Powershell commands

===========
STEP 0: Before we Start:

1. System: It runs on a Windows 2022 Pro Server. Should not make a difference to a Win 10, I think...
2. Docker: It runs on a Docker Image. So you need Docker (I have Docker Desktop)
3. Docker Nvidia Runtime: Make Sure the Nvidia Runtime is working in Docker, you can try this with the following command:

docker info | Select-String -Pattern "Runtimes"

This should give you the following output in the CLI:

Runtimes: io.containerd.runc.v2 nvidia runc

If you see this, evrything is fine

5. Nvidia Drivers: I have the latest Drivers. You then can make sure that Docker sees all GPUS and can work with them. you

can download and run a little Image to check this.
Run the following command:

docker run --rm --gpus all ubuntu nvidia-smi

you should then get a output like this:

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 595.54 Driver Version: 595.79 CUDA Version: 13.2 |
+-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |

| 0 NVIDIA GeForce RTX 4080 ... On | 00000000:01:00.0 Off | N/A |
| 0% 38C P8 5W / 320W | 9845MiB / 16376MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 1 NVIDIA GeForce RTX 5070 Ti On | 00000000:07:00.0 On | N/A |
| 0% 40C P8 12W / 300W | 5808MiB / 16303MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+

6. Git: Git must be installed for the git command

===========
STEP 1: Get the Git-Repo.

Create in your file System a folder where you want to have the source of the image. i named it "fish2"
Start Powershell and go in to that folder with the cd command. Something like this:

Powershell Command:
Set-Location "A:\fish\docker3"

After that you are ready to do the command of installation guide of https://speech.fish.audio/install/#manual-docker-build

Powershell Command:
git clone https://github.com/fishaudio/fish-speech.git
This downloads the repo

Powershell Command:
cd fish-speech

===========
STEP 2: Get the Models

You need to Download the Models from here:
[(https://huggingface.co/fishaudio/s2-pro/tree/main)]

Create two folders in your fish-speech folder:
-> checkpoints
-> references

In the Checkpoints folder you need to create a subfolder "s2-pro"
In this folder you copy all the files you downloaded from huggingface.

===========
STEP 3: Lets add some patches!
The orginal form of the app is not able to be splitt into two gpus.
So we gonna change that.
I added all the patched files so you can just replace those files.

patch.zip

===========
Patch 1:
we do changes to the following file: tools/run_webui.py

Add two new CLI arguments and wire them through:

parser.add_argument("--decoder-device", type=str, default=None)
parser.add_argument(
"--max-seq-len",
type=int,
default=4096,
help="Cap KV-cache and causal-mask allocation (default 4096).",
)

Then in main:
decoder_device = args.decoder_device or args.device

decoder_model = load_decoder_model(
config_name=args.decoder_config_name,
checkpoint_path=args.decoder_checkpoint_path,
device=decoder_device, # ← was hardcoded to args.device
)

inference_engine = TTSInferenceEngine(
llama_queue=llama_queue,
decoder_model=decoder_model,
compile=args.compile,
precision=args.precision,
llama_device=args.device, # ← new
)

llama_queue = launch_thread_safe_queue(
...
max_seq_len=args.max_seq_len, # ← new
)

===========
Patch 2:
we do changes to the following file: fish_speech/inference_engine/init.py

Root bug: send_Llama_request was passing device=self.decoder_model.device (cuda:1) as the generation device to the llama model (which is on cuda:0). This made all tensors
land on the wrong GPU.

def init(self, ..., llama_device: str = None):
...
self.llama_device = llama_device or str(decoder_model.device)

In send_Llama_request, change:

Before (wrong — uses decoder's device):

device=self.decoder_model.device,

After:

device=self.llama_device,

===========
Patch 3:
we do changes to the following file: fish_speech/inference_engine/vq_manager.py

Codes produced by the llama model (cuda:0) need to be moved to the decoder (cuda:1) before decoding:

def decode_vq_tokens(self, codes):
codes = codes.to(self.decoder_model.device) # ← add this line
return self.decoder_model.from_indices(codes[None])[0].squeeze()

===========
Patch 4:
we do changes to the following file: fish_speech/models/text2semantic/llama.py

BaseTransformer.forward_generate — normalize all inputs to the model's device at function entry:

def forward_generate(self, inp, input_pos=None, audio_masks=None, audio_parts=None, return_all=False):
dev = self.codebook_embeddings.weight.device
inp = inp.to(dev)
if input_pos is not None: input_pos = input_pos.to(dev)
if audio_masks is not None: audio_masks = audio_masks.to(dev)
if audio_parts is not None: audio_parts = audio_parts.to(dev)
# ... rest of function unchanged

DualARTransformer.forward_generate_fast — same treatment:

def forward_generate_fast(self, x, input_pos=None):
dev = self.fast_embeddings.weight.device
x = x.to(dev)
if input_pos is not None: input_pos = input_pos.to(dev)
# ... rest of function unchanged

===========
Patch 5:
we do changes to the following file: fish_speech/models/text2semantic/inference.py

decode_one_token_ar — normalize all tensor args to model device:

dev = next(model.parameters()).device
x = x.to(dev)
input_pos = input_pos.to(dev)
temperature = temperature.to(dev)
top_p = top_p.to(dev)
semantic_logit_bias = semantic_logit_bias.to(dev)
if audio_masks is not None: audio_masks = audio_masks.to(dev)
if audio_parts is not None: audio_parts = audio_parts.to(dev)
if previous_tokens is not None: previous_tokens = previous_tokens.to(dev)

generate_long — move audio_masks/audio_parts to the llama device after encoded.to(device):

encoded = encoded.to(device=device)
if audio_masks is not None: audio_masks = audio_masks.to(device=device)
if audio_parts is not None: audio_parts = audio_parts.to(device=device)

init_model and launch_thread_safe_queue — thread through max_seq_len:

def init_model(checkpoint_path, device, precision, compile=False, max_seq_len=None):
model = DualARTransformer.from_pretrained(
checkpoint_path, load_weights=True, max_length=max_seq_len
)

def launch_thread_safe_queue(..., max_seq_len=None):
def worker():
model, decode_one_token = init_model(..., max_seq_len=max_seq_len)

===========
STEP 4: Build Command
After we applied the patches, we should be able to build the Docker Image with the following command:

docker compose --profile webui build

===========
STEP 5: Launch command
When the build got succsessfull trough, we can start the docker image with the following arguments:

docker compose --profile webui run --rm --service-ports webui --half --device cuda:0 --decoder-device cuda:1 --max-seq-len 4096

===========
Results:

• No device mismatch errors
• cuda:0 usage: ~10.3 GB (down from OOM at 15+ GB)
• Generation speed: ~10–11 tokens/sec

so, yeah, thats it. hope i can safe some headache of someone...

[jedik07]

Can I run Fish Speech on an RTX 5070 Ti?

[Jan21021]

im not sure, but i think there is no 5070 TI with 32 gb ram. so your will most likely also have only 16gb ram and that won't be enough! :(