TL;DR: What do you use your mixed multi-GPU setup for? I got kinda the best of the worst specs for LLMs, with mixed professional GPU brands and mixed large memory kits. The past year, I migrated from an NVIDIA 4500 Ada 24GB to an AMD Radeon Pro W7800 32GB because I wanted to get out of the NVIDIA Linux driver hell. But then months later I realized that I have made a mistake when choosing the W7800 instead of the R9700. And then I got myself an AMD Radeon AI Pro R9700 32GB, just before the memory crisis started, But now I had 2 expensive cards just sitting on the shelf, and with no buyers because I'm in a third world country. Funny enough, the cards are like 3 years old, but now they are worth more than when I got them. Days ago, I bought an m-ATX motherboard (because space constraints) that didn't have a brain dead PCIe arrangement like all other consumer boards that expect the world to have only 3 slot GPUs (which are a waste of silicon and copper IMO). This motherboard has two x16 PCIe ports with one port in between from each other, which is perfect to connect my R9700 and Ada 4500 (sadly, the W7800 was too big for my case and CPU cooler, it will stay on the shelf for longer). The connection would be PCIe5x16 for one card and PCIe4x8 if both cards are connected (both CPU PCIe, no chipset middleman). MSI B760M Project Zero (Gen5x16+Gen3x1+Gen4x4) vs CWWK CW-W680-NAS (Gen5x16/Gen4x8+Gen3x4+Gen4x8+Gen3x4) I tried running Qwen 3.8 at Q6_K, but it reached only 30tg vs 40tg with only the R9700. And Gemma 4 QAT at 32tg vs 50tg on the R9700. Finally, Qwen Flash Next Q4_K_M was only 9tg vs 14tg on the R9700+moe-cache-mib (unusable speed either way). I could try bigger quants, but token generation is already bad on these quants. I get consistent results using layer split. With tensor split I get very mixed results, sometimes better, sometimes worse. The cards definitely don't play well together. I don't think I'm that constrained by bandwidth (at most 2GiB/s in nvtop vs max throughput of 11.45GiB/s). But maybe I'm constrained by the slower Ada 4500, or by PCIe latency. What else could this setup be used for? Running bigger models doesn't seem like the right path. Maybe setting up master/slave agents for software development or research? (any recommendations please). For those curious about my setup: Motherboard: CW-W680-NAS (m-ATX) PSU: NZXT 1000W CPU: 13th Gen Intel Core i9-13900K GPU 1: AMD Radeon AI PRO R9700 (32GB) GPU 2: NVIDIA RTX 4500 Ada Generation (24GB) GPU 3 (Not connected): AMD Radeon PRO W7800 (32GB) Memory: G.Skill Ripjaws S5 144GB@4200MHz (2x48GB@4800MHz + 2x24GB@5200MHz) Engine: Llama.cpp (vulkan, compiled myself) Config: device = Vulkan1,Vulkan2 main-gpu = 0 # R9700 tensor-split = 4,3 # 32GB/24GB, for moe-cache-mib n-gpu-layers = 999 n-cpu-moe = 999 n-gpu-layers-draft = 999 parallel = 1 # I'm the only user threads = -1 prio = -1 # attempt to avoid screen stutters batch-size = 16384 # idk what im doing here, but this gets best PP ubatch-size = 2304 # big image read crash below this flash-attn = true fit = off ctx-size = 131072 context-shift = true moe-cache-mib = 12000 # with one card, ~16000 with two   submitted by   /u/LMasterGame [link]   [comments]