I have a machine with a bunch of GPUs attached to it. 2x5070tis, 2x7900xtxes and 2x5060tis 16GB. So I did this little test to see how it fares running Qwen 3.8 Flash Next Q4_XL against my little Strix Halo. Not well. Not well at all. The full numbers are below but the high context number sums it up. @160,000 context Pile of GPUs 215.73(PP) and 16.29(TG) Strix Halo(Gufo) 1227.12(PP) and 22.04(TG) Here's the number for a Strix Halo fork of llama.cpp, Halo Box. Strix Halo(Halo Box) 587.99(PP) and 21.24(TG) Lastly, here's the mainline llama.cpp number. Strix Halo(llama.cpp 0.4.1) 113.48(PP) and 7.06(TG) For running QFN, Strix Halo really shines. Pile of GPUs Device 0: NVIDIA GeForce RTX 5070 Ti, compute capability 12.0, VMM: yes, VRAM: 15880 MiB Device 1: NVIDIA GeForce RTX 5070 Ti, compute capability 12.0, VMM: yes, VRAM: 15880 MiB Device 2: NVIDIA GeForce RTX 5060 Ti, compute capability 12.0, VMM: yes, VRAM: 15888 MiB Device 3: NVIDIA GeForce RTX 5060 Ti, compute capability 12.0, VMM: yes, VRAM: 15888 MiB ggml_cuda_init: found 2 ROCm devices (Total VRAM: 49120 MiB): Device 0: AMD Radeon RX 7900 XTX, gfx1100 (0x1100), VMM: no, Wave Size: 32, VRAM: 24560 MiB Device 1: AMD Radeon RX 7900 XTX, gfx1100 (0x1100), VMM: no, Wave Size: 32, VRAM: 24560 MiB | model | size | params | backend | ngl | fa | dev | lm | test | t/s | | ------------------------------ | ---------: | ---------: | ---------- | --: | --: | ------------ | ---------: | --------------: | -------------------: | | qwen4exp A3B Q4_K - Medium | 103.68 GiB | 176.94 B | CUDA,ROCm | -1 | 1 | CUDA0/CUDA1/ROCm0/ROCm1/CUDA2/CUDA3 | none | pp2048 | 242.13 ± 1.39 | | qwen4exp A3B Q4_K - Medium | 103.68 GiB | 176.94 B | CUDA,ROCm | -1 | 1 | CUDA0/CUDA1/ROCm0/ROCm1/CUDA2/CUDA3 | none | tg128 | 32.09 ± 0.06 | | qwen4exp A3B Q4_K - Medium | 103.68 GiB | 176.94 B | CUDA,ROCm | -1 | 1 | CUDA0/CUDA1/ROCm0/ROCm1/CUDA2/CUDA3 | none | pp2048 @ d10000 | 242.96 ± 1.12 | | qwen4exp A3B Q4_K - Medium | 103.68 GiB | 176.94 B | CUDA,ROCm | -1 | 1 | CUDA0/CUDA1/ROCm0/ROCm1/CUDA2/CUDA3 | none | tg128 @ d10000 | 30.23 ± 0.14 | | qwen4exp A3B Q4_K - Medium | 103.68 GiB | 176.94 B | CUDA,ROCm | -1 | 1 | CUDA0/CUDA1/ROCm0/ROCm1/CUDA2/CUDA3 | none | pp2048 @ d20000 | 249.34 ± 0.65 | | qwen4exp A3B Q4_K - Medium | 103.68 GiB | 176.94 B | CUDA,ROCm | -1 | 1 | CUDA0/CUDA1/ROCm0/ROCm1/CUDA2/CUDA3 | none | tg128 @ d20000 | 28.84 ± 0.05 | | qwen4exp A3B Q4_K - Medium | 103.68 GiB | 176.94 B | CUDA,ROCm | -1 | 1 | CUDA0/CUDA1/ROCm0/ROCm1/CUDA2/CUDA3 | none | pp2048 @ d40000 | 253.95 ± 1.44 | | qwen4exp A3B Q4_K - Medium | 103.68 GiB | 176.94 B | CUDA,ROCm | -1 | 1 | CUDA0/CUDA1/ROCm0/ROCm1/CUDA2/CUDA3 | none | tg128 @ d40000 | 26.29 ± 0.07 | | qwen4exp A3B Q4_K - Medium | 103.68 GiB | 176.94 B | CUDA,ROCm | -1 | 1 | CUDA0/CUDA1/ROCm0/ROCm1/CUDA2/CUDA3 | none | pp2048 @ d80000 | 252.48 ± 1.20 | | qwen4exp A3B Q4_K - Medium | 103.68 GiB | 176.94 B | CUDA,ROCm | -1 | 1 | CUDA0/CUDA1/ROCm0/ROCm1/CUDA2/CUDA3 | none | tg128 @ d80000 | 22.34 ± 0.07 | | qwen4exp A3B Q4_K - Medium | 103.68 GiB | 176.94 B | CUDA,ROCm | -1 | 1 | CUDA0/CUDA1/ROCm0/ROCm1/CUDA2/CUDA3 | none | pp2048 @ d160000 | 215.73 ± 0.56 | | qwen4exp A3B Q4_K - Medium | 103.68 GiB | 176.94 B | CUDA,ROCm | -1 | 1 | CUDA0/CUDA1/ROCm0/ROCm1/CUDA2/CUDA3 | none | tg128 @ d160000 | 16.29 ± 0.03 | Strix Halo running Gufo | model | size | backend | test | t/s | | -------------------------------- | ---------- | ---------- | ------------------ | --------------------- | | Qwen3.8 Flash Next | 103.69 GiB | ROCm (HIP) | pp2048 | 1603.47 ± 0.00 | | Qwen3.8 Flash Next | 103.69 GiB | ROCm (HIP) | tg128 | 26.53 ± 0.00 | | Qwen3.8 Flash Next | 103.69 GiB | ROCm (HIP) | pp2048 @ d10000 | 1377.97 ± 0.00 | | Qwen3.8 Flash Next | 103.69 GiB | ROCm (HIP) | tg128 @ d10000 | 25.44 ± 0.00 | | Qwen3.8 Flash Next | 103.69 GiB | ROCm (HIP) | pp2048 @ d20000 | 1353.19 ± 0.00 | | Qwen3.8 Flash Next | 103.69 GiB | ROCm (HIP) | tg128 @ d20000 | 25.01 ± 0.00 | | Qwen3.8 Flash Next | 103.69 GiB | ROCm (HIP) | pp2048 @ d40000 | 1328.64 ± 0.00 | | Qwen3.8 Flash Next | 103.69 GiB | ROCm (HIP) | tg128 @ d40000 | 24.08 ± 0.00 | | Qwen3.8 Flash Next | 103.69 GiB | ROCm (HIP) | pp2048 @ d80000 | 1283.77 ± 0.00 | | Qwen3.8 Flash Next | 103.69 GiB | ROCm (HIP) | tg128 @ d80000 | 23.00 ± 0.00 | | Qwen3.8 Flash Next | 103.69 GiB | ROCm (HIP) | pp2048 @ d160000 | 1227.12 ± 0.00 | | Qwen3.8 Flash Next | 103.69 GiB | ROCm (HIP) | tg128 @ d160000 | 22.04 ± 0.00 | Strix Halo running Halo Box Device 0: AMD Radeon 8060S Graphics, gfx1151 (0x1151), VMM: no, Wave Size: 32, VRAM: 128000 MiB | model | size | params | backend | ngl | fa | lm | test | t/s | | ------------------------------ | ---------: | ---------: | ---------- | --: | --: | ---------: | --------------: | -------------------: | | qwen4exp A3B Q4_K - Medium | 103.68 GiB | 176.94 B | ROCm | -1 | 1 | none | pp2048 | 811.79 ± 19.09 | | qwen4exp A3B Q4_K - Medium | 103.68 GiB | 176.94 B | ROCm | -1 | 1 | none | tg128 | 23.95 ± 0.03 | | qwen4exp A3B Q4_K - Medium | 103.68 GiB | 176.94 B | ROCm | -1 | 1 | none | pp2048 @ d10000 | 729.60 ± 38.99 | | qwen4exp A3B Q4_K - Medium | 103.68 GiB | 176.94 B | ROCm | -1 | 1 | none | tg128 @ d10000 | 22.46 ± 0.43 | | qwen4exp A3B Q4_K - Medium | 103.68 GiB | 176.94 B | ROCm | -1 | 1 | none | pp2048 @ d20000 | 718.14 ± 33.92 | | qwen4exp A3B Q4_K - Medium | 103.68 GiB | 176.94 B | ROCm | -1 | 1 | none | tg128 @ d20000 | 22.45 ± 0.18 | | qwen4exp A3B Q4_K - Medium | 103.68 GiB | 176.94 B | ROCm | -1 | 1 | none | pp2048 @ d40000 | 700.55 ± 33.17 | | qwen4exp A3B Q4_K - Medium | 103.68 GiB | 176.94 B | ROCm | -1 | 1 | none | tg128 @ d40000 | 22.29 ± 0.23 | | qwen4exp A3B Q4_K - Medium | 103.68 GiB | 176.94 B | ROCm | -1 | 1 | none | pp2048 @ d80000 | 646.85 ± 29.06 | | qwen4exp A3B Q4_K - Medium | 103.68 GiB | 176.94 B | ROCm | -1 | 1 | none | tg128 @ d80000 | 21.95 ± 0.25 | | qwen4exp A3B Q4_K - Medium | 103.68 GiB | 176.94 B | ROCm | -1 | 1 | none | pp2048 @ d160000 | 587.99 ± 30.10 | | qwen4exp A3B Q4_K - Medium | 103.68 GiB | 176.94 B | ROCm | -1 | 1 | none | tg128 @ d160000 | 21.24 ± 0.31 | Strix Halo running llama.cpp 0.4.1 Device 0: AMD Radeon 8060S Graphics, gfx1151 (0x1151), VMM: no, Wave Size: 32, VRAM: 128000 MiB | model | size | params | backend | ngl | fa | lm | test | t/s | | ------------------------------ | ---------: | ---------: | ---------- | --: | --: | ---------: | --------------: | -------------------: | | qwen4exp A3B Q4_K - Medium | 103.68 GiB | 176.94 B | CUDA,ROCm,Vulkan | -1 | 1 | none | pp2048 | 362.84 ± 6.19 | | qwen4exp A3B Q4_K - Medium | 103.68 GiB | 176.94 B | CUDA,ROCm,Vulkan | -1 | 1 | none | tg128 | 20.11 ± 0.36 | | qwen4exp A3B Q4_K - Medium | 103.68 GiB | 176.94 B | CUDA,ROCm,Vulkan | -1 | 1 | none | pp2048 @ d10000 | 321.75 ± 0.91 | | qwen4exp A3B Q4_K - Medium | 103.68 GiB | 176.94 B | CUDA,ROCm,Vulkan | -1 | 1 | none | tg128 @ d10000 | 19.88 ± 0.34 | | qwen4exp A3B Q4_K - Medium | 103.68 GiB | 176.94 B | CUDA,ROCm,Vulkan | -1 | 1 | none | pp2048 @ d20000 | 285.78 ± 0.42 | | qwen4exp A3B Q4_K - Medium | 103.68 GiB | 176.94 B | CUDA,ROCm,Vulkan | -1 | 1 | none | tg128 @ d20000 | 18.35 ± 0.46 | | qwen4exp A3B Q4_K - Medium | 103.68 GiB | 176.94 B | CUDA,ROCm,Vulkan | -1 | 1 | none | pp2048 @ d40000 | 237.23 ± 1.13 | | qwen4exp A3B Q4_K - Medium | 103.68 GiB | 176.94 B | CUDA,ROCm,Vulkan | -1 | 1 | none | tg128 @ d40000 | 15.48 ± 0.61 | | qwen4exp A3B Q4_K - Medium | 103.68 GiB | 176.94 B | CUDA,ROCm,Vulkan | -1 | 1 | none | pp2048 @ d80000 | 173.80 ± 0.18 | | qwen4exp A3B Q4_K - Medium | 103.68 GiB | 176.94 B | CUDA,ROCm,Vulkan | -1 | 1 | none | tg128 @ d80000 | 11.18 ± 0.11 | | qwen4exp A3B Q4_K - Medium | 103.68 GiB | 176.94 B | CUDA,ROCm,Vulkan | -1 | 1 | none | pp2048 @ d160000 | 113.48 ± 0.23 | | qwen4exp A3B Q4_K - Medium | 103.68 GiB | 176.94 B | CUDA,ROCm,Vulkan | -1 | 1 | none | tg128 @ d160000 | 7.06 ± 0.17 |   submitted by   /u/fallingdowndizzyvr [link]   [comments]