Vulkan support

[elgorkad2024]

Love the asset! Full transparency I am fairly new to the domain. For our use case GPU enablement is needed to get required responsiveness. The "Num GPU Layers" option brilliantly enables NVIDA GPUs and best I can tell use of Vulkan would get us to be able to leverage AMD and Intel GPUs too.

Is that on the roadmap? Suggestions on other options?

[amakropoulos]

Hi! LLMUnity supports CUDA (Nvidia), HIP (AMD) as well as Vulkan (fallback).

[elgorkad2024]

oh! ty, I didn't catch that. Is it default behavior and by specifying num GPU layers to a non-zero, positive value does it automatically figure out what to load/execute?

[amakropoulos]

yes, if the GPU layers are non-zero it loads the backend in the following order until a backend works:

• cuda
• hip
• vulkan

[elgorkad2024]

That's simply outstanding, thanks for the clarification and a truly impressive solution!!!
Is there anything I should include in our build or game deployment files to support all 3 backends?
Is there any try/catch or other construction I should incorporate to support if a load fails?

[amakropoulos]

no everything is handled internally :)