I've had a Strix Halo box (Radeon 8060S, 128 GB unified) for a while. vLLM runs on it, but slowly, because the stock ROCm kernels are written with MI300 in mind and not this APU. Long-context prefill was the worst of it. A 256k prompt on Qwen3.6-35B-A3B took more than two hours. So I spent the last few weeks writing HIP kernels just for gfx1151 and wrapped them in a vLLM plugin. It's called libstrix. How it works You pip install it into the same env as your vLLM. There's no fork and you don't patch anything. It hooks in through vLLM's normal plugin system. It only turns on with ROCm + gfx1151. On any other GPU it does nothing. Each kernel is built for one specific shape and refuses anything else. If a layer doesn't match, vLLM just uses its own kernel. In my testing nothing got slower than stock, single stream or batched. It covers MoE GEMMs, attention prefill and decode, the Qwen GDN layers, and the int8 / int4 / MXFP4 linears. STRIX_DISABLE=all turns it off, so you can A/B it against stock yourself. Numbers Stock upstream vLLM vs. the same vLLM with the plugin, single stream, one APU. Every cell is stock → libstrix in tok/s. "Weights" is the checkpoint's format and "KV" is the KV cache dtype. model weights KV prefill @2k prefill @64k decode @2k decode @64k Qwen3.6-35B-A3B bf16 bf16 1,583 → 3,581 260 → 2,617 17.8 → 32.6 12.8 → 26.9 bf16 int8 1,331 → 3,589 132 → 2,558 17.6 → 32.8 12.5 → 29.3 Qwen3.6-35B-A3B int8 bf16 1,824 → 3,214 263 → 2,485 29.3 → 29.3 18.2 → 25.0 int8 int8 1,504 → 3,221 133 → 2,432 29.0 → 29.7 17.6 → 27.0 Qwen3.8-27B bf16 bf16 514 → 677 55 → 533 4.3 → 4.4 3.5 → 4.0 bf16 int8 423 → 709 38 → 539 4.3 → 4.4 3.6 → 4.1 Qwen3.8-27B + MTP int8 bf16 304 → 694 50 → 537 3.3 → 7.7 2.8 → 6.1 int8 int8 270 → 692 37 → 532 3.3 → 7.8 2.8 → 6.6 Qwen3.8-27B (Quark AWQ) int4 bf16 380 → 690 47 → 534 12.8 → 12.9 7.6 → 8.8 int4 int8 335 → 688 37 → 528 12.8 → 13.0 8.2 → 10.1 LFM2.5-24B-A2B bf16 bf16 3,208 → 5,007 304 → 3,615 18.0 → 38.3 16.3 → 31.5 bf16 int8 2,530 → 5,124 311 → 3,540 17.8 → 38.3 14.5 → 31.7 gemma-4-26B-A4B bf16 bf16 1,548 → 3,591 140 → 2,108 22.8 → 24.6 12.1 → 21.2 bf16 int8 890 → 3,877 75 → 2,336 20.3 → 25.9 7.1 → 23.5 gemma-4-26B-A4B AWQ int4 bf16 1,356 → 3,395 138 → 2,225 32.6 → 37.2 14.6 → 29.7 int4 int8 867 → 3,358 76 → 2,151 28.1 → 38.2 7.8 → 33.3 gemma-4-31B QAT int4 bf16 245 → 542 31 → 389 9.0 → 10.4 5.0 → 8.1 int4 int8 179 → 535 17 → 382 8.2 → 10.8 3.1 → 9.2 gemma-4-31B MXFP4 bf16 289 → 492 33 → 397 1.6 → 9.4 1.4 → 7.4 MXFP4 int8 203 → 542 19 → 389 1.6 → 9.7 1.2 → 8.3 Some absolute numbers with the plugin at 128k context and int8 KV: Qwen3.6-35B-A3B: 1,751 tok/s prefill, 26 tok/s decode LFM2.5-24B-A2B: 2,405 tok/s prefill, 27 tok/s decode gemma-4-26B-A4B AWQ: 1,499 tok/s prefill, 30 tok/s decode At 256k, the 35B now prefills in about 4 minutes (241 s). Stock vLLM took 7,568 s. Accuracy I measured KLD against stock vLLM. On the Qwen models it's at the noise floor (0.001–0.006, 97–99% top-1 agreement). GSM8K with the plugin on vs. off shows no significant difference on any model I tested. Gemma-4-26B drifts more (KLD around 0.2). Gemma is unusually sensitive to tiny bf16 rounding differences. What it doesn't do Decode isn't always faster. On the 27B bf16, the 27B int4 and the 35B int8, decode is basically the same as stock, because none of the kernels I replace are the bottleneck there. The big decode gains show up where they are. The MXFP4 decode number is inflated. Stock vLLM runs MXFP4 through a slow emulation path on this GPU, so I'm beating a weak baseline there. It's gfx1151 only (the one I tested), not tested on other GPUs. You have to build it against the same torch and ROCm that your vLLM uses, or it fails at import. Repo: https://github.com/asf0/libstrix (MIT)   submitted by   /u/asfbrz96 [link]   [comments]