DISCLAIMER Much of this post was written by my Hermes agent. I wrote the abstract. I had it collect and format the metrics for me. But I found all of this fascinating and surprising enough that I thought it might inspire some of you gearheads out there too. Abstract Rented a few instances on vast.ai to do some benching. Blew about $40 in all on the benchmarking. The machines Box Instance GPUs CPU RAM PCIe per card $/hr b7642 58586826 2x RTX 3090 AMD EPYC 7642, 96c/192t, 2 sockets 504 GB Gen4 x8 2.3244 g3090 58586211 4x RTX 3090 AMD EPYC 7K62, 96c/192t, 2 sockets 251 GB Gen4 x16 0.7928 The runtimes and models Using Strata (up to unsloth Q8) and Maya (GLM 5.3F and DSV4.1F) at various model sizes. The results in TL;DR Don't even consider > C=1 for Maya. Strata? From > 100 tok/s down to just over 50 tok/s (at Q_8). The part that blew my mind and made me want to post: You can get almost linear concurrent stream performance scaling up to at least 8 streams on QFN 3.8 *Q8** with this build.* Workstation and server builds are worth it: you don't have to fight so much for VRAM; a couple of decent GPUs and you're off to the races. Instead it's DRAM (which sucks a whole lot less to buy--even though it still sucks) and CPUs (which aren't too bad at all) GLM and DSV: they'd likely benefit from more VRAM. But, again, 1.5k+ for a 3090, paying at least what they sold for 6-7 years ago? It's insulting. DISCLAIMER: Totally not an advertisement. Like buying a car, if I can, I want to take it for a spin before I buy. Vast.ai is amazing this way. If you can find a machine spec that you're even the least bit curious about, you lease some time on it (paying a bit extra for storage), download the runtime and models you want to test, and fire up the benches. Two definitions of aggregate (read before the tables) Aggregate = sum of per-stream tok/s (user's convention). Each stream's own rate, added up. Answers "how fast is one user's stream, and how many like it". Measured aggregate = all completion tokens / block wall time (first token of any stream to the last token of any stream). Answers "how many tokens per second does the box actually emit". They diverge because a per-stream rate counts only from that stream's first token, while the measured window also contains the time each stream spends queued. On the Maya engine the sum barely exceeds the single-stream rate (the box's total output is roughly fixed and extra streams split it). On Strata/QFN the sum grows much faster than the measured rate because the streams overlap only partially (the engine queues). Runtimes Runtime Version Models Strata terminal 0.1.55 Qwen3.8-Flash-Next IQ3_S (qwen3.8-flash-next-iq3_s) Project Maya v1.1.45, maya engine GLM-5.3-Flash Maya-S24 / Maya-L Project Maya v1.1.45, maya-ds4 engine DeepSeek-V4.1-Flash-1017 Maya-XS (IQ1_S) / Maya-S (IQ2_XXS) 1. QFN IQ3_S, Strata 0.1.55, 2x3090 (b7642) 4000-token prompt, 256 generated, no-think: C tok/s per stream aggregate tok/s (sum) measured aggregate tok/s TTFT per stream (s) wall (s) 1 121.94 122 122 3.49 9.08 2 120.31, 132.16 252 108 2.34, 3.33 4.00 4 121.60, 124.04, 121.69, 123.85 491 112 3.20, 5.07, 2.16, 4.14 5.92 6 120.13, 115.04, 119.70, 119.18, 120.32, 120.01 714 114 4.61, 13.48, 2.21, 9.04, 11.27, 6.82 15.71 8 203.84, 166.43, 206.26, 206.30, 164.32, 206.29, 205.60, 206.42 (draft-assisted) 1565 (draft-inflated; not meaningful) 183 (draft-assisted) 12.15, 3.90, 6.85, 8.17, 2.18, 10.82, 5.52, 9.49 13.41 Single stream (engine tuned: --spec 4 --kv int8 --kv-resident 32768 --ple-io ram, context 131072): Run decode tok/s (client / engine) prefill tok/s (engine) TTFT (s) notes short, thinking 133.49 / 139.80 — 0.909 short, no-think 123.70 / 118.20 — 0.418 decode @24K 114.26 / 122.90 — 6.841 prefill 4K 114.78 / — 1501.4 2.098 8-token output prefill 32K 93.38 / — 1773.3 13.229 8-token output prefill 128K 111.19 / — 1990.6 23.097 8-token output decode @128K warm 124.83 / 119.20 — 22.876 cache_n 81920 decode @128K cold 123.0 / 111.9 — 22.731 prompt_ms 22731.6 Reading: QFN holds ~110-125 tok/s per stream at C=2..6, so the sum climbs to 714 while the measured rate stays ~108-114 — streams overlap only partially. The C=8 row is draft-assisted on repetitive text and is an upper bound. 2. GLM-5.3-Flash Maya-S24, Project Maya, 2x3090 (b7642) 4000-token prompt, 256 generated: C tok/s per stream aggregate tok/s (sum) measured aggregate tok/s TTFT per stream (s) wall (s) 1 35.51 36 36 4.85 20.5 2 15.41, 12.26 28 24 9.36, 4.85 25.98 4 6.13, 4.78, 5.54, 6.87 23 20 9.43, 18.40, 4.94, 13.91 52.06 6 4.40, 5.19, 3.56, 3.80, 4.77, 4.08 26 21 18.86, 28.01, 5.14, 9.71, 23.44, 14.28 77.31 8 2.75, 2.89, 3.22, 4.13, 3.05, 3.87, 3.41, 3.62 27 22 5.08, 9.58, 18.57, 36.49, 14.07, 32.01, 23.05, 27.52 98.43 Single stream matrix: Run decode tok/s (client / engine) prefill tok/s (engine) TTFT (s) notes short, thinking 33.20 / 36.60 50.9 1.496 draft 208/254, finish=length short, no-think 37.54 / 37.10 50.1 1.344 draft 212/254 decode @24K 36.54 / 41.60 539.6 16.028 cache_n 16384, prompt_n 1843 prefill 4K 25.38 / 32.70 697.2 4.619 8-token output, cache_n 0 prefill 32K 27.05 / 31.40 1002.9 20.392 cache_n 16384, prompt_ms 7877 prefill 128K 33.11 / 32.50 1183.5 73.299 cache_n 81920, prompt_ms 12754 decode @128K 35.90 / 38.40 1173.6 73.700 cache_n 81920, draft 223/254 Reading: the sum plateaus at 23-28 tok/s from C=2 up — barely above the single-stream rate. Per-stream collapses 35 -> 3.4. GLM-5.3-Flash Maya-L (156 GB) was started and skipped by user decision; partial single-stream samples ran ~13.9-14.3 tok/s (expert cache 63-65% hit). 3. DeepSeek-V4.1-Flash-1017 Maya-XS (IQ1_S, 277 GB), Project Maya maya-ds4, 2x3090 (b7642) Drafting: kept. Setup calibration 31.6 (no draft) vs 49.9 (DSpark); serve self-calibration 33.8 vs 37.6 tok/s (+12%). C tok/s per stream aggregate tok/s (sum) measured aggregate tok/s TTFT per stream (s) wall (s) 1 44.86 45 45 6.37 18.4 2 27.19, 18.55 46 33 12.29, 6.36 21.70 4 9.47, 12.24, 7.78, 16.93 46 31 12.43, 18.33, 6.53, 24.29 39.46 6 5.21, 12.43, 6.74, 7.96, 5.66, 9.70 48 31 6.70, 35.76, 18.32, 24.18, 12.49, 30.01 56.41 8 3.87, 5.89, 8.08, 4.23, 6.82, 4.66, 5.20, 9.69 48 30 6.60, 30.06, 41.87, 12.43, 35.98, 18.32, 24.23, 47.76 73.55 Single stream matrix: Run decode tok/s (client / engine) prefill tok/s (engine) TTFT (s) notes short, thinking 26.86 / 40.90 — 1.603 draft 138/192 short, no-think 32.60 / 42.80 — 1.096 draft 88/121 decode @24K 38.35 / 55.60 — 16.402 cache_n 17730 prefill 4K 40.24 / 40.10 196.5 7.083 cache_n 2581 prefill 32K 41.56 / 42.70 197.7 19.632 cache_n 23787 prefill 128K 45.24 / 52.90 185.1 68.985 cache_n 96519 decode @128K warm 42.37 / 60.00 183.3 69.155 cache_n 96521, draft 180/207 4. DeepSeek-V4.1-Flash-1017 Maya-S (IQ2_XXS, 339.5 GB), Project Maya maya-ds4, 2x3090 (b7642) Drafting: disabled by calibration — 21.6 (no draft) vs 22.2 (DSpark) tok/s sustained; engine runs --dspark none. C tok/s per stream aggregate tok/s (sum) measured aggregate tok/s TTFT per stream (s) wall (s) 1 29.87 30 30 7.82 24.2 2 18.60, 11.95 31 24 15.88, 8.22 29.65 4 11.28, 6.72, 8.48, 5.57 32 22 31.72, 16.27, 24.23, 8.34 54.41 6 7.92, 4.65, 4.08, 6.45, 5.40, 3.64 32 22 46.53, 23.72, 15.95, 39.13, 31.43, 8.14 78.85 8 6.72, 3.45, 4.26, 3.15, 3.81, 2.89, 4.85, 5.63 35 23 59.37, 23.05, 37.30, 15.72, 30.16, 8.26, 44.69, 52.01 97.49 Single stream matrix: Run decode tok/s (client / engine) prefill tok/s (engine) TTFT (s) notes short, thinking 17.57 / 26.60 40.9 2.753 short, no-think 24.51 / 27.10 44.8 2.029 decode @24K 23.25 / 28.90 127.2 19.403 cache_n 17729 prefill 4K 30.55 / 32.90 131.4 7.740 cache_n 2582 prefill 32K 29.92 / 30.70 128.6 20.797 cache_n 23790 prefill 128K 27.60 / 29.50 121.6 72.781 cache_n 96514 decode @128K 24.05 / 25.60 118.0 72.781 cache_n 96520 Cross-arm reading Sum of per-stream rates vs single-stream rate: S24 36 -> 23-28 (flat); Maya-XS 45 -> 46-48 (flat); Maya-S 30 -> 31-35 (flat). The Maya engines add essentially nothing in aggregate between C=1 and C=8: extra streams split the same fixed output. QFN is the exception — its sum grows (122 -> 714 at C=6) because its per-stream rates do not degrade — yet its measured wall-clock rate still sits at ~108-114, because the streams only partially overlap. Size a box against the measured aggregate; size a user's experience against the sum (and against the per-stream column). Drafting decisions differ by quant: XS keeps the DSpark drafter (+12% on the serve's own calibration), S drops it (22.2 vs 21.6 — not faster). Only QFN/QFN-class clears the "2-3 streams at >=50 tok/s each" bar; every 300B-class Maya arm fails it at C>=2. C=1 wall times include stream setup; the C>=2 walls are the full ladder block windows. Unsloth QFN rungs on the same box (Q4 -> Q8_0 sweep) Same machine, same runtime (Strata), same driver (uxl_rung.sh), same bench harness — only the GGUF rung changes. Per-stream values are the individual stream rates; "sum" is the arithmetic total at that C; "measured" is the client wall-clock aggregate. Full per-run matrix tables live in the case page. Quant on-disk C=1 C=2 (sum / measured) C=4 C=6 C=8 verdict UD-Q4_K_XL 111.3 GB 89.14 197 / 96.50 413 / 100.74 618 / 94.60 817 / 99.33 clears 50 tok/s per stream at every C UD-Q5_K_XL 158.3 GB 81.51 178 / 86.90 333 / 80.73 488 / 74.92 675 / 82.19 clears it at every C, ~10-15% below Q4 UD-Q6_K_XL 169.2 GB 71.44 146 / 70.65 407 / 95.52 460 / 72.74 589 / 70.81 clears 50 tok/s per stream at every C (one mis-run: these rows first landed in the log labelled q8) Q8_0 188.2 GB 94.20 121 / 56.07 233 / 56.39 363 / 58.04 471 / 56.89 straddles the line — most streams 54-62, two samples under 50 (46.2 at C=2, 48.4 in the short_think matrix run) Note the same pattern as every other arm on this box: the sum keeps climbing while the measured aggregate stays in a ~75-100 band, because the streams only partially overlap.   submitted by   /u/sleight42 [link]   [comments]