I originally installed llama cpp via their install script on their page; it used Vulkan by default. The GPU (NVIDIA) would "disappear" and I'd have to restart the machine. I re-built llama cpp for CUDA, and had to remove that original install, but the llama cpp built with/for CUDA is much more stable. If I also get an AMD GPU and want to have both in my PC, do I also have to build a separate llama cpp "instance" for HIP usage? I'm not planning on using the same instance of llama cpp with the NVIDIA and AMD card at the same time (not trying to tensor split or parallelize the same LLM across those GPUs, or whatever). I might have one model running on one GPU, a second model running on the second GPU, for different purposes, not always simultaneously, so in that sense, would it make sense to keep the current version of llama for CUDA and build a second version of it that I could alias as llama-hip (different ports, etc)? If any of you have done this, I'd love to read your thoughts. If it does make sense to split a model in a 1:2 ratio (16GB CUDA:32GB HIP), I'm happy to experiment, but that wasn't my intention. Initial LLM chat shows that I could try something like this: ```sh Do you need a separate binary for HIP? Yes — a HIP build is a distinct build. HIP is not "added at runtime" to your CUDA binary; it has to be compiled in with -DGGML_HIP=ON, using the ROCm toolchain. The good news: the two builds can coexist happily. Just use separate build directories: your existing NVIDIA build stays as-is cmake -B build-cuda -DGGML_CUDA=ON cmake --build build-cuda --config Release -j a sibling HIP build HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)" \ cmake -B build-hip -DGGML_HIP=ON -DGPU_TARGETS=gfxXXXX -DCMAKE_BUILD_TYPE=Release cmake --build build-hip --config Release -j ``` The rest was basically warnings about why trying to use the same instance of llama cpp (built for both CUDA and HIP) could be problematic, ending with: sh What I'd recommend 1. Keep your CUDA build for the 5060 Ti. Add a separate HIP build for the Radeon. Two binaries, two build dirs. 2. Run two server instances (different ports), one per card. Best stability, best per-card performance, and you use both GPUs simultaneously. 3. Treat "one model spread across CUDA+HIP" as experimental — try it if curious (build-both, -sm layer -ts 16,32, --list-devices), but don't build your workflow on it. If a single model genuinely needs >32GB and won't fit on the Radeon alone, that's the case where a mixed build is the only option short of Vulkan — and it may not deliver the memory benefit due to the duplication bug.   submitted by   /u/espece-de-bon [link]   [comments]