My SXM2 Game is Impeccable. I overcome obstacles. This room temperature is always tropical. TL;DR: I'm running DeepSeek-V4.1-Flash for a single user on four 2017-era V100 32 GB cards plus system RAM. Generation runs 36-38 tok/s from 8K to 75K context with thinking off, and 32-38 tok/s with thinking on. Prefill is about 1,000-1,200 tok/s from 4K to 128K depth. The context window is set to 524,288 tokens. It's built on the 1Cat-vLLM fork (at its #1277 merge), which does the hard Volta (sm70) work. I just built, tuned and benchmarked it. Hardware CPU: AMD Threadripper PRO 3945WX (12C/24T), single NUMA node RAM: 256 GB DDR4 (8x 32 GB), run at 2667 MT/s for these tests GPUs: 4x Tesla V100 SXM2 32 GB on one SXM2 carrier board NVLink between all four (2 links per pair) Host side: PCIe Gen3 x8 per GPU, behind a PLX switch with a single Gen3 x16 uplink to the CPU Clocks/power: SM clock locked at 1522 MHz. Power limit 150 W per card, raised to 175 W for the API-speed and llama-benchy runs Storage: the main GGUF loads from a SATA 8 TB hard drive. That only affects boot (about 22 minutes to load). The Engram file is on NVMe and gets pre-read into page cache. Software / config 1Cat-vLLM at the #1277 merge (vLLM 1.5.2.dev1277), Python 3.12, PyTorch 2.10 + CUDA 12.8, NVIDIA driver 580 Tensor parallel 4, max-num-seqs 1 (single user), full CUDA graphs (batch sizes 1-6), async scheduling, prefix caching, chunked prefill, custom all-reduce over NVLink Model: `DeepSeek-V4.1-Flash-1017-GSQ-RCO-IQ2_XXS` GGUF, 4 shards, ~232 GB Engram table: `DeepSeek-V4.1-Flash-Engram-IQ4_NL` GGUF (110.6 GB), kept in system RAM Experts: about a third of the expert weights live in pinned host RAM (`expert_cache_offload_fraction: 0.33`) and are swapped onto the GPUs on demand Speculative decoding: built-in DSpark drafter, FP8, 5 draft tokens Thinking on by default; tool calling with the `deepseek_v41` parser max-model-len 524288 Results API decode speed (single request, unique cache salt per request, so no prefix-cache hits) Test Prompt Decode tok/s (median of 3, range) TTFT Prefill tok/s Accepted tokens per draft 8K, thinking off ~8.0K 38.3 (36.6-39.0) 6.6 s 1,202 2.27 8K, thinking on ~8.0K 37.9 (36.8-39.5) 6.5 s 1,234 1.39 75K, thinking off ~75K 35.8 (33.2-38.1) 69.1 s 1,085 2.21 75K, thinking on ~75K 31.9 (31.4-34.9) 69.2 s 1,084 1.34 8K, Engram file not yet in page cache ~8.0K 25.9 (n=1) 9.9 s 808 2.34 The last row shows why it's worth pre-reading the 110 GB Engram file into RAM after boot. llama-benchy 0.4.0: prefill (pp2048, concurrency 1, `--no-cache`, thinking off, 3 runs each) Context depth pp2048 t/s Time to first response 0 578 4.0 s 4K 1,224 5.5 s 8K 1,014 10.6 s 16K 1,093 17.3 s 32K 1,117 31.6 s 64K 1,095 62.2 s 128K 1,008 132.5 s llama-benchy 0.4.0: generation (pp2048 + tg, concurrency 1) Context depth tg128 t/s tg1024 t/s 0 28.5 24.9 1K 24.8 31.2 4K 28.9 20.0 With a 5-token drafter, generation speed depends on how well the drafter guesses. llama-benchy's prompts are random book text, so tg1024 swings ±6-8 t/s from run to run. I left out benchy's tg32 numbers (13-26 t/s): 32 tokens is too short to measure anything with speculative decoding. Engine-level numbers (offline, no HTTP) Warm decode round (every expert the round needs is already on the GPUs): 27.8 ms, against 24.9 ms in upstream's reference. In the profiler a round spans 29.5 ms, of which the GPUs are busy 24.9 ms. That matches upstream, so the kernels perform as expected. The extra ~5 ms is host-side: CPU work at the start of each round, a blocking device-to-host copy, and small kernel-launch gaps. Newer upstream commits target exactly this per-step host overhead. Fresh text (expert-cache misses): 116.7 ms/round and 25.0 tok/s, against upstream's 53.5 ms and 54.5 tok/s. Caveat: I ran this with the Engram file cold, so it isn't apples to apples. A warm rerun is on my list. Host-to-GPU copies: 6.5 GB/s per GPU on its own, but 12.9 GB/s total (3.2 GB/s each) when all four copy at once, because they share one Gen3 x16 uplink. In the trace, all four GPUs are swapping at the same moment about 76% of the swap time. Correctness Greedy (temperature 0) output is identical run to run within one load (8 prompts x 2). After a restart, 13 of 16 greedy outputs matched exactly. The 3 that diverged flipped at near-ties (top-two margins of 0.06-0.375 nats). I haven't tracked down the cause yet. Thinking on/off and tool calls work. All 8 upstream micro-tests pass (two needed hard-coded paths in the test files changed). What's next PCIe switch change: move to two switches so each Gen3 x16 uplink feeds 2 GPUs instead of 4. That roughly doubles host-to-GPU bandwidth when all cards swap experts at once. My estimate from the trace is a modest ~1.5 ms/round (~5%) on warm text; it should matter more on fresh text. RAM to 2933 MT/s (it was dropped from 3200 after memory errors). Pull the newer 1Cat commits (per-step host fixes and DeepSeek-V4.1 updates) and re-run the same suite. Move the GGUF from the hard drive to NVMe for faster boots. Credits All credit for the Volta DeepSeek-V4.1 path (GGUF experts, expert swapping, DSpark drafting on sm70) goes to the 1Cat-vLLM project (GitHub: yangzhuxinyzx and contributors). Happy to answer questions or share the full config.   submitted by   /u/TumbleweedNew6515 [link]   [comments]