This has made a massive improvement in performance on my 7900XTX before: | model | size | params | backend | ngl | n_ubatch | fa | test | t/s | | ------------------------------ | ---------: | ---------: | ---------- | --: | -------: | --: | --------------: | -------------------: | | gemma4 26B.A4B Q4_0 | 13.26 GiB | 25.23 B | Vulkan | -1 | 1024 | 1 | pp512 | 3410.53 ± 22.72 | | gemma4 26B.A4B Q4_0 | 13.26 GiB | 25.23 B | Vulkan | -1 | 1024 | 1 | tg128 | 135.64 ± 0.80 | | gemma4 26B.A4B Q4_0 | 13.26 GiB | 25.23 B | Vulkan | -1 | 1024 | 1 | pp512 @ d8192 | 2477.27 ± 76.78 | | gemma4 26B.A4B Q4_0 | 13.26 GiB | 25.23 B | Vulkan | -1 | 1024 | 1 | tg128 @ d8192 | 120.42 ± 0.46 | | gemma4 26B.A4B Q4_0 | 13.26 GiB | 25.23 B | Vulkan | -1 | 1024 | 1 | pp512 @ d16384 | 2130.36 ± 35.16 | | gemma4 26B.A4B Q4_0 | 13.26 GiB | 25.23 B | Vulkan | -1 | 1024 | 1 | tg128 @ d16384 | 118.42 ± 0.14 | after: | model | size | params | backend | ngl | n_ubatch | fa | test | t/s | | ------------------------------ | ---------: | ---------: | ---------- | --: | -------: | --: | --------------: | -------------------: | | gemma4 26B.A4B Q4_0 | 13.26 GiB | 25.23 B | Vulkan | -1 | 1024 | 1 | pp512 | 4331.21 ± 132.67 | | gemma4 26B.A4B Q4_0 | 13.26 GiB | 25.23 B | Vulkan | -1 | 1024 | 1 | tg128 | 142.76 ± 0.92 | | gemma4 26B.A4B Q4_0 | 13.26 GiB | 25.23 B | Vulkan | -1 | 1024 | 1 | pp512 @ d8192 | 2885.67 ± 80.24 | | gemma4 26B.A4B Q4_0 | 13.26 GiB | 25.23 B | Vulkan | -1 | 1024 | 1 | tg128 @ d8192 | 125.29 ± 0.14 | | gemma4 26B.A4B Q4_0 | 13.26 GiB | 25.23 B | Vulkan | -1 | 1024 | 1 | pp512 @ d16384 | 2402.13 ± 44.16 | | gemma4 26B.A4B Q4_0 | 13.26 GiB | 25.23 B | Vulkan | -1 | 1024 | 1 | tg128 @ d16384 | 120.82 ± 0.18 |   submitted by   /u/nickm_27 [link]   [comments]