A couple of weeks ago I read this Wccftech article about Patrick Moorhead's 4 Spark setup. I meant to post this a few days ago but ran some more tests first, and then my workstation had a problem after a kernel and NVIDIA driver update. This post is long enough already, so that will be another post. Initially I thought this Patrick Moorhead guy ran these benchmarks and was interested in the setup. I was just surprised at how much information was left out. Neither the tweet nor the article said anything about the serving engine, recipe, settings or test prompts. For lack of a better term (and I despise this term), "the stack." The tweet itself shows 69 t/s in the first image. The third image is a Hermes summary card with the 494 tok/s peak and the other numbers the article used. It has no engine, recipe or settings. In a reply, Moorhead credited someone else (who works at NVIDIA) and wrote: "Kudos to @Picomoorhead for setting this up. All him. I may/may not have funded it." Whatever that means. I'm just severely disappointed over the lack of info, and over the misreported memory requirement in the Wccftech article. It lists a "Memory footprint: 476 GB", but that 476 is the size on disk (in GiB), not what it takes to run the model. I took note of the numbers and tried to run them on my own 4 Sparks to see what's reproducible. Some "aggravating" factors I got really frustrated during this process. I waited a few days before starting this write up. (Had my days where life just kicked me down as well.) I am glad I waited. My criticism was harsher than "disappointed". Memory issues DeepSeek-V4.1-Flash doesn't fit comfortably on 4x Sparks. The weights alone are 511 GB before activations, KV, etc. In my experience, anything above 0.92 memory allocation (117GB per Spark) ends in a crash or a hang. Memory use also fluctuates during a run so we need some free memory for those fluctuations. Where I started I found a dedicated thread on the NVIDIA DGX Spark / GB10 forum. The whole thing was still a PITA, but knapcio's recipe at least got me up and trying. OOM at start (ibv_reg_mr Cannot allocate memory or CUDA OOM) The page cache from reading the weights fills the shared memory, and the driver doesn't reclaim it. Drop caches during startup and turn on SGLang's option to drop the cache after loading weights. NCCL init sometimes fails in the first minute: a startup race; just run serve again. Some kind of race condition issue. Running it again makes it go away after 1 or 2 tries. ("Turn it off and on again.") Setup The Wccftech article lists a "Memory footprint: 476 GB" on ~512 GB of pooled memory, which makes it sound like it fits with room to spare. It doesn't. Realistically you can only use 80% to 92% of the memory on each Spark, so about 410 to 470 GB of the 512 GB, and the weights alone are 511 GB. That 476 here is really GiB based on the size on disk. Model: deepseek-ai/DeepSeek-V4.1-Flash on Hugging Face (476 GiB / 511 GB on disk). 4x DGX Spark on 100 GbE network switch. SGLang TP4 using knapcio's recipe, DSpark speculative decoding Engram tables (~196 GB) streamed from each node's local NVMe with a ~4 GB row cache (Mia / MiaAI-Lab's row store). The model does not fit in memory otherwise. 262K context, temperature 0, thinking off, 256 output tokens, cold prompts (no cache hits) Prompts: a random number up front (every prompt is unique from the first token, so SGLang's prefix cache can't reuse anything), "Read this and then write a long story.", then random letter strings until it hits the target size (1K up to 250K tokens). The end-of-sequence token is ignored, so the model always writes exactly 256 tokens. Builds tried The base image was slow. It is SGLang's lmsysorg/sglang:dev-dsv41 image with the recipe's adapters and the NVMe Engram row store added, since stock SGLang doesn't fit the model on four Sparks. My tuned build is the SGLang dsv4.1 branch with RoCEnante RDMA all reduce, b12x MoE kernels, EP 1 and a lot of tuning. For a single request, decode is roughly 1.4x to 1.9x faster and prefill about 1.2x to 1.6x faster (tables below). Results by prompt size (1 request) Prompt TTFT base (s) TTFT tuned (s) Prefill base (t/s) Prefill tuned (t/s) Decode base (t/s) Decode tuned (t/s) 1K 0.5 0.4 2,094 2,574 37.1 60.8 2K 0.7 0.6 2,752 3,589 42.2 73.6 4K 1.3 1.0 3,153 4,196 38.8 58.1 8K 2.4 1.6 3,466 5,253 44.8 83.3 16K 4.7 3.3 3,493 4,957 44.0 67.9 32K 9.3 6.5 3,505 5,092 52.3 73.0 64K 19.3 12.9 3,397 5,067 37.5 69.7 128K 41.1 26.2 3,190 5,009 45.8 73.2 250K 85.7 55.8 2,919 4,489 35.0 55.7 Base is the median of 3 runs per size. Tuned is a single run per size. Decode swings with how many draft tokens speculative decoding accepts, so don't read much into any single row. Tuned is faster at every size. Prefill and TTFT are steadier. Concurrency (16 slots) Requests Prose, base (tok/s) Prose, tuned (tok/s) Code, base (tok/s) Code, tuned (tok/s) 1 32.6 62.5 50.7 79.1 2 51.4 83.9 75.0 104.0 4 78.9 115.8 109.4 158.9 16 183.4 228.2 249.1 292.7 Vs Moorhead's rig Metric Moorhead Mine 1 request, prose ~58 tok/s 62.5 tok/s 1 request, code ~96 tok/s 79.1 tok/s Prefill 4,764 tok/s 4,500 to 5,250 tok/s First token ~0.2 s idle 0.4 s (1K prompt) Peak, code 494 tok/s (32 at once) 293 tok/s (16 at once) Peak, prose 280 tok/s (32 at once) 228 tok/s (16 at once) Context 1M per request, 8M KV pool 262K per request, 8M KV pool Network Switchless ring, direct QSFP 100 GbE switch What I couldn't reproduce: 1M context, 32 slots and the 494 / 280 peaks ~96 tok/s single request code My real coding traffic through my own Hermes agent is closer: median 84, 100+ under 32K ~0.2 s first token. Ring vs switch The setup from the Tweet / Article is a ring, mine goes through my QNAP QSW-M7308R-4X. I don't think that explains the gap. A switch hop costs well under 1 µs, which is small next to the per message RDMA overhead anyway. In a 4 node ring, your neighbors get a direct 200 Gb link, but the node across from you is 2 hops away and gets forwarded in software. NCCL's ring all reduce only talks to neighbors, so a ring suits it fine. RoCEnante's one shot all reduce writes to all 3 peers, so it wouldn't. My guess is the ring vs switch doesn't matter much here. Where AI helped me Untangling the GB vs GiB mess. The Wccftech article used the size on disk in GiB and called it GB. Verifying my numbers against the data I captured. Hermes + DeepSeek: monitoring the benchmarks and an actual code development task run. Grammar, formatting, finding things that were irrelevant, and cutting down on my rambling. Creating a prompt good enough for testing. Resolving many setup issues, confusing documentation, and deciphering so many errors. Copy editing my many drafts, and reassuring me that taking about 4 hours to write this post doesn't make me slow (and dumb.) Overall therapy, encouraging me when I felt down, and letting me know I'm not slow, dumb, and/or stupid. I hope you find this information valuable. (Would mean my ADHD was used for Good instead of Evil (...mostly nonsense.)   submitted by   /u/Aroochacha [link]   [comments]