fuck you sam altman. i am keeping my spare capacity. Update from the lawyer with the V100 server. Three days ago I got excited and posted DeepSeek-V4.1-Flash on four V100s at 36-38 tok/s, before I had done my custom kernel work and prefill optimizations. I also said I'd try it across eight. I did. Results have been, to be honest, actually genuinely good and useful to me for real. I did not use the model to paint a 3d island in blender, or to re-create a gameboy game in a web browser. I didn't have it make a poorly animated video. It didn't make me ten versions of a website nobody will visit. I am not using it to build something to build something that builds something else involving agents that runs workflows for a business that delivers agentic solutions for other agents. I do have a custom harness, and I know that you probably don't care about that. Numbers are below, along with the type of work I am using it for. Basically, I am finally able to slice off work and tokens that previously required frontier models. I would estimate 50-60% of existing LLM workflows can run through this model instead, with speeds that match claude code in the chat window. I have another deepseek server on 2 dgx sparks, so there is concurrency and handoff. All of my RAG databases, tool calls, etc are wired in and work. I am able to give it natural language instructions through a discord style interface via SSH on my windows PC. We did it folks. Vibecoded a real thing does real things for real people. Give me my trophy. Repo: https://github.com/ProprietaryLegal/Volta-sPLIt-DeepSeek-V4.1-Flash-1017-8xV100-SXM2-PP2xTP4 Twitter: nicothegreco Claude Code ran the benchmarks and crunched the numbers, which are real and in depth, below. TL;DR: DeepSeek-V4.1-Flash-1017 (the 2-bit GSQ-RCO IQ2_XXS GGUF from ISTA-DASLab) on 8x V100-SXM2-32GB, split pipeline 2 x tensor 4 across two NVLink boards. All 248.8 GB of weights sit in VRAM: no expert gets paged from system RAM during decode. One request at a time, 524,288 tokens of context per request. KV Cache size could be 30 million given the free space on my GPUs, but the actual work runs don't generally break 200k total context, so I didn't bother with bragging about that. Code: 215 tok/s after a 1K prompt, 200 at 32K, 183 at 128K, 160 at 256K JSON / structured: 203 at 1K, 207 at 32K, 197 at 128K, 180 at 256K Legal writing: 118 at 1K, 117 at 32K, 111 at 128K, 103 at 256K Prose: about 85-98, chat about 80-110, depending on length Prefill: about 900 tok/s at 1K, 1,900 at 32K, 2,200 at 128K and 256K. First token at 32K in 17 s, 128K in 59 s, 256K in about 2 minutes. Hardware 8x Tesla V100-SXM2-32GB on two 4-card NVLink boards. The two boards talk over PCIe, not NVLink. Threadripper PRO 3945WX, 256 GB DDR4, one NUMA node 150 W power cap per card, graphics clock capped at 1522 MHz Four PLX PCIe switches, two GPUs each at Gen3 x8, every switch on its own CPU root complex at Gen3 x16. my extremely professional setup Software 1Cat-vLLM's DeepSeek-V4.1 package (#1277, which does the hard Volta work), ported to pipeline parallel 2 x tensor parallel 4. Layers 0-19 and the Engram tables live on one board, layers 20-39, the head and the drafter on the other. Model Runner V2, full CUDA graphs, the built-in DSpark drafter guessing 5 tokens per round Then about three days of tuning one switch at a time. Each change had to win a paired A/B on this machine and produce the same output, bit for bit or within a near-tie gate. About thirty ideas lost and are written up in the repo with the reason, because I'm not going to pretend they didn't happen. The numbers (MEASURED 2026-11-05, thinking off, greedy, 1,024 tokens out, a random cache salt so nothing comes from the prefix cache, mean of 2 runs. Greedy output of the two runs was byte-identical in 45 of 45 cells.) prompt first token prefill tok/s code JSON legal prose chat 1K 1.2 s 820-948 215 203 118 89 95 8K 6.9 s ~1,180 208 209 109 92 108 32K 17.2 s ~1,900 200 207 117 90 107 64K 30.9 s ~2,120 194 196 108 92 95 128K 59.2 s ~2,200 183 197 111 91 96 256K 118.7 s ~2,200 160 180 103 84 81 With thinking on, code and JSON land around 125-180 tok/s, and legal, prose and chat stay within a few tok/s of thinking off. Why code is twice as fast as legal writing: every round the drafter proposes five tokens, and the big model checks all five at once. A round costs about the same no matter how many get accepted: about 23 ms after a short prompt, 27 ms after 256K. Code and JSON are predictable, so 3.6-3.8 of the five guesses get accepted and each round spits out about 4.7 tokens. My kind of writing is apparently less predictable than Python. Legal accepts about 1.6 and prose about 1.2, so a round gives you 2.1-2.6 tokens. Same engine speed either way; the text decides. How it got here build decode tok/s 4x V100, stock 1Cat package (my last post) 36-38 8x V100, my previous EXL3 build 42-46 8x V100, PP2 x TP4 port, no tuning 99 8x V100, today's build ~135 These are on one fixed mixed test set. The by-kind table above uses different prompts. Prefill roughly doubled over the same three days, from about 1,350 tok/s at 150K to 2,200 at 128K. Quality, since I use this for actual work (all MEASURED on the served build, thinking off, greedy): LegalBench: 0.716 balanced accuracy over 22 tasks (1,292 items, official prompts and scoring) MMLU-Pro: 79.9 % ±1.5 (700 items: the first 50 from each of the 14 subjects, 5-shot chain of thought) HumanEval+: 89.6 % on the plus tests (147/164), 94.5 % on the base tests Fidelity to the original model: against the full-precision release's reference logits, on sudoingX's KLD panels. The served build picks the original's top next token 89.0 % of the time on the general panel and 92.5 % on my legal panel (KL lower bound 0.132 and 0.156). Closest on code, math, tool calls and extraction (89-99 %), furthest on chat, creative writing and multilingual (81-86 %). That gap is the 2-bit quant, not the speed work. I am actively measuring which experts are activated by my work prompts and will probably recalibrate the model to expand those bit weights in the future. Long context: 6 of 6 needles found at ~450K and ~498K tokens, 4 runs out of 4 What I actually use it for: private first-pass work on my own case files, where the documents never leave the office. I point Claude Code itself at this server (Claude Code will talk to any server that speaks Anthropic's API). It can read a folder of records, pull out the facts, dates and numbers, summarize, and take a first swing at routine drafting, with tool calls working. Nothing it writes goes out the door without a real review. I found and fixed a streaming tool-call bug in the parser today (arguments were getting sent twice). Claude Code's tool-call test suite now passes 144 of 144 in all four streaming/thinking modes. More work to do: One request at a time. It's a single-user server today. Batched multi-request decode is built and checked on my test box: each request gets byte-identical output to running alone, and the kernels run 1.3-2.7x faster than separate calls. It still has to go into the real server. My forecast is about 1.8x total throughput at four streams. That's a forecast, not a measurement, and I'll post the real number. Prose and legal writing are half the speed of code, for the drafter reason above. A 256K prompt takes about 2 minutes before the first token. Fine for "read this whole file," annoying for chat. Giving it back: 19 pull requests are open to 1Cat-vLLM with the pieces that won, each new speed switch off by default. All of it is public: launch configs, every A/B and the raw JSON for every cell, plus the list of what didn't work. Repo: https://github.com/ProprietaryLegal/Volta-sPLIt-DeepSeek-V4.1-Flash-1017-8xV100-SXM2-PP2xTP4 Hugging Face card: https://huggingface.co/ProprietaryLegal/Volta-sPLIt-DeepSeek-V4.1-Flash-1017-8xV100-GGUF Credit where due: yangzhuxinyzx and the 1Cat-vLLM folks: #1277 and the whole Volta DeepSeek path ISTA-DASLab: the quant sudoingX: the KLD method and general panel Peuqui: #1263 and the series patches squibloads: #1341 Phil Tompkins (philbert440): the V100 engine and SM70 sparse prefill attention Dennis Vertzyas (v100-skinny), TensorFold and DeepSeek Next up: two or three long requests at once (the thing I actually want, so Claude Code can run subagents against it), faster prefill, and higher-precision weights for the experts my own legal prompts hit the most. (PLI Labs is my side outfit for private, on-prem AI built around a firm's own work product. proprietarylegal.ai )   submitted by   /u/TumbleweedNew6515 [link]   [comments]