Strix Halo int4 cores: Gemma 4 31B at 2.43x llama.cpp, same quality

Wait 5 sec.

The 8060S has an int4 x int4 WMMA instruction at 2x the FP16 rate (we measured 104 TOPS). Nothing in ROCm or llama.cpp uses it. We wrote a GEMM for it and run Gemma 4 with int4 weights and int4 activations. To maintain accuracy we use a combination of rotation and GPTQ, along with selective mixed precision. llama-bench, Gemma 4 31B QAT Q4_0 (llama.cpp b11056 Vulkan, -fa 1). Server benchmark, 2k-token prompts, greedy, single stream: pp512 pp2048 decode with MTP llama.cpp 287 276 ours 679 724 Full test sets, same settings, 31B: GSM8K IFEval MMLU-Pro llama.cpp 96.2 92.2 ours 96.4 91.7 Run it: curl -fsSL https://embedl.com/install.sh | sh && export PATH="$HOME/.local/bin:$PATH" embedl login embedl serve google/gemma-4-31B-it/ryzen-ai-max-395 -d embedl infer google/gemma-4-31B-it/ryzen-ai-max-395 --prompt "Explain mixture-of-experts in one sentence." embedl bench google/gemma-4-31B-it/ryzen-ai-max-395 12B, E4B and E2B are up too (2.80x / 1.58x / 1.39x end to end, in the chart). Swap the name, e.g. google/gemma-4-12B-it/ryzen-ai-max-395. Numbers and methodology: https://www.embedl.com/google/gemma-4-31B-it/ryzen-ai-max-39   submitted by   /u/No-Dragonfly6246 [link]   [comments]