Update from the lawyer with the V100 server. I went and got two DGX Sparks, because of course I did. The mid-life crisis remains on schedule. Same deal as before: still a lawyer, still driving the whole thing through Claude Code, and Claude Code also ran the benchmarks, built these tables and helped me write this up. I'm Nick Mermiges, Esq., founder of PLI Labs. The repo below is ours. The setup: DeepSeek-V4.1-Flash-1017 at 2.75 bpw EXL3, vLLM tensor parallel across two Sparks. The twist is a third box, my EPYC with the 3090s, doing two jobs for the Sparks: - it holds the Engram tables in system RAM and serves them over RDMA; - it runs the DSpark speculative drafter on an NVLinked pair of 3090s. The memory that frees on the Sparks goes to KV cache: about 1.93M tokens of pool. There's also a prefix cache on the Sparks' NVMe, so a 150K prompt survives a server restart. First token comes back in 2.58 s instead of 58.6 s cold. Numbers. I re-implemented MiaAI's sparkDash protocol so this would be apples to apples with the most popular 2x Spark build. Sparks at 2300 MHz, 3 reps, medians, today. Test 1 stream 4 streams (total) Structured output (JSON) 138 tok/s 322 tok/s Code 110 tok/s 192 tok/s Prose, temp 0, thinking off 66 tok/s 130 tok/s Prose, thinking on (temp 0.7) 56 tok/s 105 tok/s Prefill (sparkDash method): Prompt tok/s 4K 2,827 32K 2,667 128K 2,650 256K 2,618 (first token in 100 s) For comparison: - MiaAI publishes 2,065–2,155 prefill and bertholomus 2,439–2,568 from 8K to 128K, so prefill is where this build wins. - Code at 4 streams is 192 against MiaAI's 166. Where I lose, because I'm not going to pretend: - MiaAI is faster on plain prose: 80.7 vs my 66 at one stream. - Decode after a 128K prompt is 62 tok/s against bertholomus's 99. - A short request stuck behind a cold 93K prompt waits 34 s. That's slightly better than bertholomus on the same test (38.6 s). MiaAI reports 3.8 s on a different version of it, and that's the thing I most want to fix. The drafter loves code and JSON (4–5 tokens accepted per round) and hates prose (about 2). Accuracy, since I use this for actual work: on sudoingX's KLD panel, real tool-call turns agree with the original model's top token 95.9% of the time (KLD 0.018). bertholomus publishes 79.4% on their own tool-call set, and MiaAI doesn't publish one. sudoingX's newer 2.8 bpw pack is a hair closer than mine on the same panel (0.0175 vs 0.0184), so credit where due. The weights are a legal-weighted allocation of HeNryous's 2.75 bpw build. All the credit for the quant goes to them. The code, launch scripts, the bench tool and the raw JSON for every cell are all public. The only exception is the prefix-cache code, which goes up in the next push. - Repo: https://github.com/ProprietaryLegal/DeepSeek-V4.1-Flash-2xDGX-Spark-x86-offload - Raw data: the runs/ folder in the repo Still to come: - the full 1M-token context test; - llama-benchy tables at depth; - BFCL tool calling; - a head-to-head on the same KLD test with the other packs. The 100 Gb cables go in Monday, so the Spark-to-3090 link gets faster too. Tell me what I'm doing wrong. Seriously. (PLI Labs is my side outfit for private, on-prem AI built around a firm's own work product. proprietarylegal.ai if that's your thing. That's the whole pitch.)   submitted by   /u/TumbleweedNew6515 [link]   [comments]