Skip to content

Pick Intel XPU by default when there is no CUDA, keep flash_attn to CUDA - #96

Open
EvilFreelancer wants to merge 1 commit into
salute-developers:mainfrom
EvilFreelancer:xpu-device-default
Open

EvilFreelancer wants to merge 1 commit into
salute-developers:mainfrom
EvilFreelancer:xpu-device-default

Conversation

@EvilFreelancer

Copy link
Copy Markdown

What

load_model(device=None) picks cuda if available, otherwise cpu, so on a machine with an Intel GPU (PyTorch xpu backend) GigaAM silently runs on the CPU unless the caller passes device="xpu". The model itself already works on XPU as it is: GigaAMASR.forward autocasts on the model's own device type, and nothing else is CUDA-specific.

This PR:

  • adds _default_device(): cuda, then xpu (when torch.xpu exists and reports a device), then cpu;
  • extends the existing flash_attn fallback from "disable on CPU" to "disable on anything but CUDA", since flash_attn is a CUDA-only library;
  • adds tests/test_device.py (9 tests, no model download): the default-device order including a torch build without torch.xpu, an explicit device being kept, and the flash_attn policy per device type.

Tested

On an Intel Arc A770 (xe driver, torch 2.10.0+xpu, Ubuntu 24.04 container):

  • pytest tests/test_device.py: 9 passed;
  • gigaam.load_model("v3_e2e_rnnt") without device loads onto xpu:0;
  • long-form transcription of a 712 s Russian recording (40 VAD chunks, batched RNNT decoding, fp16 encoder) gives the same text word for word as an RTX 5060 Ti with torch 2.10.0+cu128, in about the same time (14.5 s against 13.7 s).

Nothing changes for CUDA or CPU users.

load_model(device=None) chose cuda or cpu, so an Intel GPU (torch xpu) was only used when asked for explicitly, although the model runs there as it is: the forward already autocasts on the model's device type. The default is now cuda, then xpu, then cpu. flash_attn is a CUDA-only library, so the existing CPU fallback now covers every non-CUDA device.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant