https://github.com/bpx03/QEngine QEngine was built for 3x RTX 3090s. I was able to compile for RTX 5090s. I've now run Qwen3.8-Flash-Next on the same box through four different engines. QEngine (bpx03, v1.3.1) is the fastest for single-stream decode by a clear margin, level with Strata on agentic prompt latency, and it has a KV-in-RAM mode that makes returning to idle conversations much cheaper. Details below, including the parts that make it look less good. Rig: 2x RTX 5090 (32 GB each), Threadripper PRO 3975WX, 251 GB RAM, 262K context. QEngine's INT4 AutoRound build of Flash-Next. The README only lists sm_86, so I built it for sm_120 by changing CMAKE_CUDA_ARCHITECTURES from 86 to 120 (nvcc 13.3, g++-13). No other changes. It's up and serving 53 s after launch. Speed (club-3090 bench.sh, same script and prompts for every engine) Engine / quant Decode narrative / code (tok/s) Prefill @10K / @90K (tok/s) QEngine 1.3.1 (KV in VRAM, default) 370 / 439 ~5.7K / ~7.3K QEngine 1.3.1 (FLASH_KV=ram) 406 / 504 ~6.2K @10K Strata ISTA IQ3_S 214 / 293 ~5.1K / ~10.0K Strata UD-IQ4_XS 180 / 239 4,264 / 8,399 bucko vLLM fast-256k (tuned) 135 / 141 2,888 / 3,886 bucko vLLM ple (shipped config) 120 / 118 ~2.25K / ~3.1K Infernix (sdekok fork, TP2) 14-18 n/a Read the decode column with care. QEngine's expert cache adapts to recent use, and bench.sh repeats the same prompts, so these are close to best case. In an earlier test on varied traffic I measured about 300-400 tok/s single-stream (hermes agent traffic ran 330-470 per request). Strata's cache holds ~99.5% of routed expert mass, so repeating a prompt doesn't help it. The fair comparison is roughly 1.4-1.9x Strata and ~3x bucko. Prefill is a split result. QEngine is faster at 10K and slower than Strata at 90K, because Strata's rate climbs with depth and QEngine's stays flat. Agentic latency (12-turn tool-use bench, context grows to 35K) Turn-12 prompt read, with the prefix cache working: QEngine 1.3.1: 1.75 s Strata IQ4_XS: 1.7-1.9 s ThinkingCap 27B dual-fast: 2.07 s exl3: 3.5-3.9 s bucko: 4.36 s QEngine and Strata are level here (QEngine's rc4 build read 1.4 s in the same bench, so this one is within noise of "about 1.5-1.8 s"). Both are about 2x faster than bucko. Idle conversations: FLASH_KV=ram 1.3.1 can keep the KV cache in pinned host RAM (13.5 GB pool, about 1M tokens) instead of VRAM. I tested 8 conversations of ~58K tokens each, then 400K tokens of other traffic, then went back to each one: KV in RAM: 8 of 8 came back warm, 1.56-2.01 s to first reply. KV in VRAM: 2 of 8 warm. The other six re-prefilled in 8.7-9.4 s, about the same as a cold start (8.85 s). Answers were still correct, so you lose latency, not correctness. The RAM mode also decoded ~10% faster single-stream here, but ~10% slower with 8 concurrent 64K streams (124 vs 139 tok/s per stream). One boot per mode, one pass. Quality and stability Through the omp agent harness (44-task assistant set, effort low, one draw): 44/44 on 1.3.1, against 42/44 twice on 1.3. At medium and xhigh on 1.3: 43/44 each. A 6-task sub-agent delegation set passed 6/6 with and without delegation. The 1.3 to 1.3.1 gap is one task, so treat it as noise. Coding check (aider-polyglot-30, thinking on, effort low, single pass), on the rc4 build: QEngine 26/30. Strata UD-IQ4_XS 24/30, UD-Q4_K_XL 25/30 (24-26 across repeats), ThinkingCap 27B 17/30. 26 vs 24-26 is inside one Strata config's repeat spread, so the honest claim is that the INT4 AutoRound quant is not worse at coding, not that it's better. I haven't re-run this on 1.3.1. I also ran my usual multi-pack agent/tool-use suite earlier; QEngine landed level with the other Flash-Next engines, within noise. Bottom line On 2x5090, QEngine 1.3.1 is the fastest Flash-Next engine I've measured for single-stream decode, level with Strata on agentic prompt latency, and the KV-in-RAM mode is a real win for long idle sessions. Coding quality looks level with Strata's GGUF quants. The costs are tight VRAM, a build-from-source step, and less runtime history than vLLM or Strata.   submitted by   /u/Relative-Ant-9249 [link]   [comments]