Follow-up to my post about Strix Halo kernels for vLLM. A few of you asked for two things: an easier way to run it than building vLLM + ROCm + the plugin yourself, and Qwen3.8-Flash-Next. Both are done. Docker image docker pull ghcr.io/asf0/strix-vllm:latest It bundles my vLLM fork, the libstrix plugin and ROCm 10.1 (Ubuntu 26.04, Python 3.14), already compiled for gfx1151. You don't need ROCm on the host, only the amdgpu driver. The repo has one compose file per model I've tested: git clone https://github.com/asf0/libstrix && cd libstrix/docker/recipes cp .env.example .env # point MODELS at your model folder docker compose -f qwen3.8-flash-next.yaml up -d If the model folder isn't there, it downloads it from Hugging Face. The compile caches go to a volume, so only the first start is slow. The recipes cover Qwen3.8-Flash-Next, Qwen3.6-35B-A3B (bf16 and INT8), Qwen3.8-27B (bf16, INT8 + MTP, Quark int4, Quark MXFP4), LFM2.5-24B-A2B, and gemma-4 26B/31B (bf16, AWQ, MXFP4, QAT). I started every recipe in the image and checked that it gives a sane answer. Someone asked last time about memory overhead. Here is what each recipe gets at the default 0.85 GPU memory setting: model weights KV cache full 131k-token requests that fit Qwen3.8-Flash-Next AWQ 79.5 GiB 12.3 GiB 3.2 Qwen3.8-27B bf16 + MTP 52.3 GiB 45.1 GiB 9.1 Qwen3.8-27B INT8 + MTP 29.6 GiB 66.6 GiB 13.4 Qwen3.8-27B Quark int4 16.8 GiB 80.5 GiB 18.4 Qwen3.6-35B-A3B bf16 + MTP 70.1 GiB 21.9 GiB 12.9 Qwen3.6-35B-A3B INT8 + MTP 36.6 GiB 55.3 GiB 32.6 LFM2.5-24B-A2B 46.2 GiB 52.4 GiB 39.4 gemma-4-26B-A4B bf16 49.5 GiB 48.6 GiB 11.5 gemma-4-26B-A4B AWQ 15.6 GiB 82.1 GiB 19.4 gemma-4-31B MXFP4 17.8 GiB 79.9 GiB 4.7 gemma-4-31B QAT 18.8 GiB 75.1 GiB 4.4 Qwen3.8-Flash-Next Last time I said there's no fork. This model is the exception. Its n-gram (engram) table is 102 GB, and that doesn't fit in 128 GB next to the weights, so vLLM itself has to change. My fork (asf0/strix-vllm, branch gfx1151) streams the table from disk and prefetches the rows for the next chunk into the page cache. Everything else is still the plugin, and the other models still run on stock vLLM. You need the AWQ W4A16 checkpoint; bf16 doesn't fit. On the libstrix side, I added kernels for: the int4 MoE GEMM with int8 activations QSA sparse attention, where 16 rows share one union of selected tokens, about 1.5x Triton the QSA indexer the hyper-connection up projection Decode reads int8 copies of the bf16 linears. Same format as last time: stock → libstrix in tok/s, single stream, both on the fork, 16k prefill chunks, no MTP: model weights KV prefill @2k prefill @64k prefill @128k decode @2k decode @64k decode @128k Qwen3.8-Flash-Next AWQ int4 bf16 807 → 1,422 827 → 1,661 827 → 1,635 13.1 → 24.7 16.5 → 26.1 14.4 → 25.1 With MTP 2 on real text, libstrix does 1,539–1,617 tok/s prefill and 40–43 tok/s decode at 5.6k–100k prompts. Accuracy: KLD against stock is 0.008–0.019 with 97–98% top-1 agreement, perplexity is within 0.6%, and GSM8K is 93 vs 92. It's a bit above the other Qwens, most likely because the MoE GEMM quantizes activations to int8. Heads-up if you pulled main in the last few days The int8 decode copies (dense_w8) were turned on for every model. On Qwen3.8-27B bf16 that's about 27 GB of extra copies, and with MTP it could take the machine down. Those copies are now limited to Qwen3.8-Flash-Next, the model I measured them on, and capped at 12 GB. STRIX_ENABLE=dense_w8 turns them on elsewhere if you want to experiment. Please pull. Known issues Stock vLLM's GDN conv kernel crashes on Qwen3.8-Flash-Next, so STRIX_DISABLE=all doesn't work on that model; keep gdn_conv on. The image is a 3.6 GB download but about 14 GB once unpacked, mostly the ROCm SDK. Repos: https://github.com/asf0/libstrix (plugin, Docker recipes) and https://github.com/asf0/strix-vllm (fork).   submitted by   /u/asfbrz96 [link]   [comments]