Follow-up to my earlier post on Qwen3.8-27B decode speed: https://www.reddit.com/r/LocalLLaMA/comments/1vuli3h/ Some people asked for the recipe, it's at the bottom. That post said "lossless". I meant BF16 weights and BF16 KV, and that part holds. The outputs didn't match no speculation, though. I checked token ids, not text. vLLM nightly, DFlash2 k=9 + XQA, BF16, temp 0, one request at a time, a LoRA loaded but not used, 24 prompts, 10,496 tokens: speculation on vs off: 15 of 24 prompts differ --max-num-seqs 1 vs 32, no speculation: 21 of 24 differ a different attention backend: 20 of 24 differ So a capacity setting moved more outputs than speculation did. I haven't found why. On SGLang, deterministic mode alone isn't enough. With speculation on, 19 of 24 prompts still gave different tokens and all 24 had different logprobs. Two changes to verify fixed it: Verify runs the GDN recurrence through the decode kernel. The old verify kernel tiled the state in a different order and kept beta in fp32. Verify attention runs as decode rows, one per draft position. With both, 24 of 24 prompts match no speculation, every token id and every logprob. Same with every draft forced to reject (24 of 24), with 8 requests in flight (24 of 24), and on 3 long prompts up to 128k (3 of 3). To get the same result run after run you also need a DeepGEMM fix. Without it, 25 of 1,200 requests sent in batches of 8 came out different, and single prompts of 560 to 740 tokens did in 5 to 15% of runs. Deterministic mode was also slow: 18.0 t/s without speculation, against 26.4 stock. DeepGEMM's sm120 bf16 kernel reads the weights at about a third of DRAM bandwidth at small batch. My PR sends those GEMMs to a Triton tile that gives the same bits: https://github.com/sgl-project/sglang/pull/42145. With it, exact decode without speculation is 27.7 t/s. Decode t/s, single stream, temp 0, one RTX PRO 6000 Blackwell Max-Q: short prompts 2.2k tokens 128k tokens SGLang, no speculation 26.4 26.6 23.2 vLLM, DFlash2 k=9 + XQA (not exact) 109.8 67.4 56.1 SGLang deterministic + DFlash 8 + patches + my PR (exact) 106.6 73.9 42.6 Exact is 3% behind vLLM on short prompts, ahead at 2k, 24% behind at 128k. Prefill is slower too: 68.5 s to first token on a 128k prompt, vs 44.3 s on vLLM. I need this because I compare quants and engines on task quality. If the engine drifts by itself, I can't tell that apart from the change I'm testing. Recipe: Qwen/Qwen3.8-27B at 1d4bf0f2, BF16. Draft: incoai/Qwen3.8-27B-DFlash2 at 015e7956 SGLang main at 542a043, plus the two verify changes and PR #42145 as I measured it DeepGEMM 0.2.0 plus the sm120 fence. The upstream fix is https://github.com/sgl-project/DeepGEMM/pull/93, merged into dev on 10-03, not in a release yet all three patches with apply steps: https://gist.github.com/bill-h-lin/d2d48e99073dec4bfb81996a1440054f SGLANG_BATCH_INVARIANT_OPS_SMALL_M_TRITON=16 Launch: python -m sglang.launch_server --model-path Qwen/Qwen3.8-27B \ --enable-deterministic-inference --attention-backend triton \ --speculative-algorithm DFLASH --speculative-draft-model-path incoai/Qwen3.8-27B-DFlash2 \ --speculative-num-draft-tokens 8 --mamba-ssm-dtype float32 \ --max-running-requests 8 --chunked-prefill-size 8192 --mem-fraction-static 0.88 \ --context-length 262144 --disable-radix-cache   submitted by   /u/stargate425 [link]   [comments]