I've been keeping a list of the engines that run Qwen3.8-27B and Flash-Next on consumer hardware, mostly to see where my own engine (capy-flash) stands. Sharing it because most numbers going around aren't comparable. Our rows are measured on my box: 2× RTX 5070 Ti, 96 GB DDR5. "Verified" means a verified LocalMaxxing run or a paired run on the same machine; everything else is what the author reports. Decode is tok/s, one request. What actually makes the difference Tensor parallel vs pipeline. TP (capy-flash, SGLang, vLLM, ExLlamaV3, Paiton on AMD) adds up the bandwidth of two cards for one request. Layer split (Strata, llama.cpp) adds memory, not speed. Experts out of VRAM (Flash-Next). Hot experts in VRAM, misses computed on CPU (Strata, capy-flash, Infernix, fngine). What counts is how many experts miss per step. Quant format. NVFP4 is better quality but more bytes. 2–3-bit IQ GGUF is faster at about twice the KL (Strata's numbers: 0.144 vs 0.070). Drafters. MTP gives 1.3–1.7×. On the 27B, DFlash2 and DSpark give 2–3×: that's where most 400+ tok/s headlines come from. A DFlash2 drafter for Flash-Next came out today (incoai), so far only run on SGLang. Single-model kernels (MegaCapybara, NInfer, capy-flash) beat general engines by 2–6× at batch 1. vLLM and SGLang win once you have many users. Qwen3.8-27B Hardware Engine tok/s Verified 1× 5090 MegaCapybara 313–540 (greedy) no 1× 5090 SparkInfer (DSpark) 265 avg, 420 code no 2× 5070 Ti capy-flash (DFlash2) 379 yes 7900 XTX / R9700 hipfire / Lucebox 318 / 208–228 no 2× 5090 SGLang fork (TP2) 315 yes 1× 5090 TensorFold 220 chat, 305 code no 1× 3090 tqLLM / ExLlamaV3 273 / 242 yes 1× 5090 llama.cpp (DFlash2) 205 yes 1× 5060 Ti 16 GB tqLLM (3-bit) 156 yes 2× 5060 Ti lynx-gt (TP2) 126 no Mac M5 Pro Splash 74 no Prefill: NInfer 12.8k tok/s on a 5090 (reported), capy-flash 9.2k at 4–16k on 2× 5070 Ti without P2P (boards with P2P get ~6k for now), MegaCapybara ~7.6k. Qwen3.8-Flash-Next (125B MoE, 6B active) Hardware Engine tok/s Verified 5090 + 5060 Ti + 128 GB Basalt (NVFP4) 239–479 no 2× R9700 Paiton (TP2, W3A8 / W4A8) 272 / 240 no 1× 5090 fngine / Infernix 146–209 / 140 no 2× 5070 Ti + 96 GB Strata IQ3_S 195 yes 2× 5070 Ti + 96 GB capy-flash (NVFP4) 178 yes 1× 5090 + 128 GB strata-nvfp4 fork 177 no 2× 5090 llama.cpp fork 142 yes Mac M5 Max 128 GB MTPLX / Strata-Mac 126 / 119 no 2× 5070 Ti + 96 GB Strata NVFP4 120 yes 2× 3090 + 128 GB vLLM + CPU experts 101–111 no 5060 Ti + 5080 Strata IQ4_XS 108 yes DGX Spark ninfer-gb10 60 no 1× 5060 Ti 16 GB Strata IQ3_S / llama.cpp 48 / 30–40 no Short-prompt decode isn't the whole story. I replayed a real 53-turn coding session (growing to 81k tokens) on my box: capy-flash 177 s, Strata IQ3_S 197 s, Strata NVFP4 268 s. Prefill at 4k / 16k / 64k: capy-flash 2.4k / 4.6k / 4.8k, Strata IQ3_S 2.2k / 3.8k / 5.2k. What it costs EU street prices this month, new, VAT included: RTX 5090 ~€9.5k, R9700 €2.2k, 5070 Ti €1.3k, 5060 Ti 16 GB €740, 96 GB DDR5 €1.8k, 128 GB ~€2.4k. RAM is counted where the model needs it; CPU, board and PSU are not. Flash-Next Setup Cost Engine tok/s tok/s per €1k 2× R9700, model in VRAM €4.4k Paiton 272 62 (reported) 2× 5070 Ti + 96 GB €4.4k Strata IQ3_S 195 44 2× 5070 Ti + 96 GB €4.4k capy-flash 178 40 MacBook Pro M5 Max 128 GB ~€5.7k MTPLX 126 22 (reported) 1× 5090 + 128 GB €11.9k strata-nvfp4 fork 177 15 (reported) DGX Spark ~€6.3k ninfer-gb10 60 10 (reported) 2× 5090 €19k llama.cpp fork 142 7 27B (fits in VRAM) Setup Cost Engine tok/s tok/s per €1k 1× 5060 Ti €740 tqLLM 3-bit 156 211 2× 5070 Ti €2.6k capy-flash 379 146 2× 5060 Ti €1.5k lynx-gt 126 85 (reported) 1× 5090 €9.5k MegaCapybara / llama.cpp 440 / 205 46 / 22 2× 5090 €19k SGLang fork 315 17 At these prices two mid-range cards beat a 5090 by 3–10× per euro. The surprise for me was RAM: 96 GB of DDR5 now costs more than one of my GPUs, so on an experts-in-RAM box it's about 40 % of the bill. That puts 2× R9700 with the whole model in VRAM on top for Flash-Next, if Paiton's numbers hold up, and makes 64 GB builds worth chasing. Nobody seems to run Flash-Next on 2× 5060 Ti yet. If you have verified runs I missed, post them and I'll add them.   submitted by   /u/FeydRowan [link]   [comments]