It's been a few releases since my last post, so here is where capy-flash is now. It's my own engine for two models, Qwen3.8-27B and Qwen3.8-Flash-Next. It started as my ninfer-tp2 fork of NInfer, which added tensor parallel over PCIe, but at this point it's an engine of its own and very little of the original is left. What it's for Two RTX 50 cards on Linux. Tensor parallel across both, and for Flash-Next a VRAM expert cache shared by the two cards, with the misses computed on the CPU. Everything is tuned on one box: 2× RTX 5070 Ti 16 GB, Ryzen 9 9900X, 96 GB DDR5. It was born specifically for the NVFP4 Nvidia quantisation. It also runs on smaller setups, so more people can try it: one 16 GB card (5060 Ti and up), Flash-Next with 64 GB of RAM, mixed pairs like a 5070 Ti with a 5060 Ti. People have posted numbers from 2× 5080, 2× RTX PRO 4000 Blackwell and 2× 5060 Ti in the issues. It runs on Windows too, through WSL2 or Docker. Two cards there are slower than on Linux, and a native Windows port is in progress. What it isn't It isn't trying to beat NInfer on a single card, or Strata on a single card. If you have one GPU, or an unusual mix (a 5090 next to a 5060 Ti, a Mac, AMD cards), those are the engines to look at. They're very good, and on one card they're often faster than mine. capy-flash runs on one card mostly so you can start with one and add the second later without changing engine or model files. Where it stands All on my box, one request at a time. Decode is LocalMaxxing reasoning-v1, and "verified" means a verified LocalMaxxing run. Qwen3.8-Flash-Next Weights Cards Decode tok/s Prefill 64k tok/s NVFP4 (96 GB RAM) 2× 5070 Ti 178 4,828 3-bit (64 GB RAM) 2× 5070 Ti 234 3,994 3-bit (64 GB RAM) 1× 5070 Ti 121 2,921 All three decode numbers are verified runs. The 3-bit experts are smaller, so they decode faster. NVFP4 is the quality pick: about half the KL of the 3-bit experts by Strata's own numbers, and on GSM8K and MMLU-Pro it matches vLLM on a GB300 with the same weights. How that compares with Strata, from my own runs on the same box: NVFP4, two cards: clearly ahead. With the same NVIDIA weights I'm about 1.5× faster on short prompts and about 1.76× on average across languages and code, and a real 53-turn coding session takes 177 s against 268 s. That was against Strata 0.1.45, the last version I measured with NVFP4. 3-bit, same IQ3_S experts as Strata 0.2.2: Strata is faster overall. Long prefill is faster on Strata with one card or two. On one card its decode is faster too, but that comes mostly from its route tail skip, which drops low-ranked experts that miss the cache. I compute every expert the model picks. With the skip off, Strata decodes about 10 % slower than me on one card. On two cards my decode is a bit ahead even with the skip on. Qwen3.8-27B. Strata doesn't run it. On two cards it fits in VRAM: Weights Cards Decode tok/s Verified NVFP4, DFlash2 2× 5070 Ti 379 yes NVFP4, MTP 2× 5070 Ti 175 no NVFP4, MTP, 8 requests total 2× 5070 Ti 927 no 3-bit, MTP 2× 5070 Ti 288 yes 3-bit, MTP 1× 5070 Ti 174 yes The two rows marked "no" come from my own concurrency benchmark, not from LocalMaxxing. Prefill with NVFP4 on two cards is about 9,300 tok/s at 4-16k and 6,300 at 64k. The fastest verified two-card run I've seen on LocalMaxxing besides mine is an SGLang fork on 2× 5090 at 315. 3 bits cost something: GSM8K is the same as NVFP4, MMLU-Pro drops from 0.78 to about 0.72. With two cards NVFP4 is the better choice. What changed since v0.6 Start with one card, add the second later. Flash-Next with 64 GB of RAM on one or two cards (ISTA-DASLab's 3-bit experts, used as they are), the 27B on one 16 GB card with 3-bit weights. Context cache on disk (idea and first version by @KouShuo). A long agent session survives a restart. On the 27B with two cards, 80k tokens resume in 0.7 s instead of a 21 s prefill, 140k in 1.1 s instead of 48 s, same tokens. Faster long prompts on the 64 GB Flash-Next: 64k prefill +19 % on two cards, +35 % on one. Agent clients. Claude Code, Codex CLI and OpenCode work against the server, setup is in the README. If your client sent a developer role and got a 400, the HF artifacts now ship a chat template that accepts it. CPU side. Expert threads are picked from your CPU and respect taskset, AVX2 CPUs work, the API key can come from a file, and a second server on the same port is refused. Qwen4 The target is full Qwen4 support as soon as possible after it comes out. Flash-Next already uses most of the new architecture, so a good part of the work is done. The repo, and a favour The repo has grown a lot in the last weeks: almost 100 stars and around 30 issues and PRs. Most of the last releases came from those issues, so thanks to everyone who opened one. It's a one-person project, though. I'm happy to get issues, numbers from your machine and feature requests, but I can't keep up with all of them. Focused, well-structured PRs are very welcome and are the fastest way to get something in. One more thing. GitHub still lists capy-flash as a fork of NInfer, which it isn't anymore. If you know how to detach it without losing the history, the stars and the rest, please message me. I have tried a ticket but it didn't work and was blocked by the ai assistance. Limits Best on Linux. On Windows it runs through WSL2 or Docker, slower with two cards. RTX 50 series (sm_120). My numbers come from one machine, plus what people report in issues. On Flash-Next, one request at a time is the tested case. If you run two 50-series cards I'd like to see your numbers, especially boards with P2P and CPUs other than a 9900X. Repo: https://github.com/ValerioDolci/capy-flash Releases: https://github.com/ValerioDolci/capy-flash/releases   submitted by   /u/FeydRowan [link]   [comments]