Posted this on the r/R9700 subreddit as well, but figured I'd share my initial findings here. tl;dr - One user writes at about 190 tok/s. Prefill runs at about 6,800 tok/s total no matter how many users there are. Agent follow-ups start in 0.2 to 2.5 s and write at 80 to 225 tok/s per user, even with 8 users at 128K context each. The worst case, 8 users each sending a fresh 128K prompt at the same moment, waits about 2.5 minutes for the first token. Yes yes I know the cards aren't all matching (the perfectionist in me is annoyed but they'll be in a case so who cares). PC Specs: CPU - EPYC 7402P Motherboard - Asrock ROMED8-2T RAM - 128gb DDR4 2666mh GPU - 3x Gigabyte R9700, 1x XFX R9700. All cards power limited to 210W. PSU - Seasonic Prime TX-1600W Case - Phanteks Enthoo Pro 2 Server Fans - 3x 140mm at the front, 3x 120mm in the middle, and another 120mm for rear exhaust. I bought everything in the past two weeks, my total cost was roughly $16K CAD. The CPU/RAM/Motherboard were all bought on ebay. We'll see if the RAM is enough, but based on everything I've seen online it should be fine for my use-case. I'm close to being able to do proper inference testing, but I just ran memtests, CPU stress tests, and GPU stress tests to rule out any potential issues. Even at full blast the cards are way quieter than I anticipated, I thought it would sound like a jet engine but they're quieter than my air purifier when it's on its highest setting. Also pretty happy with the cable management, this case is so easy to build in because of how massive it is. It's also nice that it has a second area to mount additional fans. Built this as a multi-user inference server for my business. I've been using a Strix Halo 395+ 128GB for the past year and it's been incredible to see how far the community has been able to get that machine to run. Qwen 3.8 FN has been a revolution for me in my work. We deal with very sensitive data and for many of our customers, it meant using any AI inference was a huge risk. Technically we could have done it, but I wanted to err on the side of caution. Was really excited to share and post this, benchmarks are currently running for the next two hours (long context with lots of concurrency). ----------------------- ------------Benchmarks ---------------------------------------------------------- 4x Radeon AI PRO R9700 + Radiance: Qwen3.8-Flash-Next long-context concurrency numbers (32K to 128K, 1 to 8 users) Software Ubuntu 26.04.1, kernel 7.0 with the inbox amdgpu driver, ROCm 10.1 userspace Radiance v1.17.0 (codeberg.org/StillDeadcode/radiance), Docker image pinned by digest Model: qwen3.8-next-flash-fp8-iq4r-moe.rad (int8 trunk, 4-bit codebook experts, MTP drafting 3 tokens) Flags: --tp 4 --max-num-seqs 8 --max-model-len 200000 --placement expert_tiered --host-pool-mib 1024 --gpu-headroom-mib 96 --kv-cache-dtype fp8 --prefix-cache-mib 32768 --num-speculative-tokens 3 --max-num-batched-tokens 2048 At tp4 the whole model sits in VRAM (17.4 GiB of weights per card). The 47.7 GiB n-gram/PLE table is pinned in host RAM. The KV cache gets 11.6 GiB per card, which is 1.6M tokens, or 8 full 200K sessions. Method Context is real English technical text (Radiance's own docs), cut to exact token counts with the server's tokenizer Each user's prompt starts with a unique session line, so no user can reuse another user's cache All users in a round fire at the same instant (worst-case burst) Turn 1: document plus a question, cold. Turn 2: the same conversation plus the answer and a follow-up question (the agent pattern) 500 max output tokens, temperature 0.6, thinking off, streamed. First-token time and writing speed are measured on the client Writing speed per user = output tokens after the first, divided by the time after the first token Results Context Users Turn 1 first token (median / max) Turn 1 writing per user Turn 2 first token (median / max) Turn 2 cached Turn 2 writing per user Turn 2 total 32K 1 4.8 s / 4.8 s 195 tok/s 0.28 s / 0.28 s 95% 209 tok/s 170 tok/s 32K 2 9.0 s / 9.3 s 141 tok/s 0.43 s / 0.57 s 95% 156 tok/s 244 tok/s 32K 4 16.7 s / 20.1 s 69 tok/s 0.82 s / 1.21 s 95% 115 tok/s 322 tok/s 32K 8 36.9 s / 37.2 s 83 tok/s 2.22 s / 2.54 s 94% 91 tok/s 406 tok/s 64K 1 9.3 s / 9.3 s 185 tok/s 0.19 s / 0.19 s 98% 225 tok/s 194 tok/s 64K 2 18.6 s / 18.7 s 144 tok/s 0.28 s / 0.38 s 98% 157 tok/s 265 tok/s 64K 4 33.8 s / 40.9 s 52 tok/s 0.64 s / 0.74 s 98% 132 tok/s 393 tok/s 64K 8 74.4 s / 74.9 s 76 tok/s 1.30 s / 1.53 s 98% 94 tok/s 491 tok/s 128K 1 18.8 s / 18.8 s 189 tok/s 0.29 s / 0.29 s 99% 180 tok/s 149 tok/s 128K 2 37.5 s / 37.8 s 126 tok/s 0.47 s / 0.56 s 99% 131 tok/s 209 tok/s 128K 4 68.9 s / 83.4 s 36 tok/s 1.18 s / 1.42 s 99% 102 tok/s 259 tok/s 128K 8 150.8 s / 151.5 s 70 tok/s 2.20 s / 2.46 s 99% 80 tok/s 362 tok/s Short prompts for comparison (about 30 tokens in, about 500 out): one user writes at 187 tok/s. At 1 / 2 / 4 / 8 users the total is 175 / 290 / 489 / 763 tok/s, or 177 / 148 / 125 / 103 tok/s per user. What I took from it Prefill runs at about 6,800 tok/s total no matter how many users there are. 8 x 128K (1.02M tokens) took 151 s. Cold prompts that arrive together are interleaved, so everyone gets their first token at about the same time, at the end. With 8 x 128K that means about 2.5 minutes for everyone. Follow-ups are where this setup shines. 94 to 99% comes from cache, the first token arrives in 0.2 to 2.5 s even at 8 users x 128K, and each user writes at 80 to 225 tok/s. Context length barely slows writing. One user writes at 189 tok/s at 128K versus 195 at 32K. Turn 1 writing speed depends on whether other users are still prefilling. At 4 users the early finishers write while the others' prompts are still being read, so the slowest user dropped to 9 to 32 tok/s. At 8 users everyone finishes reading together and then writes together at 70 to 83 tok/s. Size the host prefix cache for users x context. In an earlier run with --prefix-cache-mib 12288 (prompt sizes not exact, some above 64K), 8 x 64K follow-ups were only 71% cached, with a 37.6 s worst first token and 9 tok/s per user. With 32768: 98% cached, 1.5 s, 94 tok/s. The README's four-rank row carries --host-pool-mib 12288 over from two ranks. On four cards the whole expert plane is resident and the pool only holds about 312 MiB per card, so 12 GiB per card pinned 48 GB of RAM for nothing and left 3.6 GB free. 1024 is plenty. Every card sat at its 210 W cap during prefill. The longest section (8 x 128K) pushed the two middle cards to 91 C hotspot and 92 C memory.   submitted by   /u/Anarchaotic [link]   [comments]