I've been messing with Qwen 3.8 Flash Next for a while on my local inference server and stumbled onto Strata, saw a small headline/article, didn't make much of it, passed on reading it fully. Then again somewhere else, then once more... So I gave it a quick read, didn't quite believe it, today I finally tested it and wow, this thing is amazing. https://preview.redd.it/otabxo1cm5th1.png?width=722&format=png&auto=webp&s=e4a9d3593c51d6c7ba15d709dd48d21f5dab78a4 About a month ago I spent about a week trying to do something vaguely similar myself and didn't get results this good, keeping the useful experts on GPU, pushing the rest to wherever made more sense, different caching/offloading ideas, etc. Promising results but not enough to use and trust. The proverbial hat is off to Niko1221, the guy pulled it off nicely. My box: 2x RTX 3090 24GB Ryzen 7 9800 64GB DDR5-5200 Linux/Proxmox Qwen3.8-Flash-Next-OrcaRouter IQ4_XS Vision enabled I'm getting a bit over 100 t/s decode on short context, and the whole machine is weirdly chill while doing it, for comparison, my prior vLLM setup was doing about 60-70 t/s on decode and made it sound like it was gonna take off (it's close to me, so it was a bit annoying). I thought "ok I'm keeping this, onto the next thing for today"... then I saw it supports experimental context extension and obviously I had to try it. So far I've actually completed: 200K prompt: ~2,004 t/s prompt processing, ~73 t/s decode 400K prompt: ~1,633 t/s prompt processing, ~46 t/s decode 900K prompt: ~1,110 t/s prompt processing, ~44 t/s decode And the screenshot is from me actually using the 512K setup right now, not necessarily saying 512K/1M is good effective context yet, but I'm test driving for today. I'm leaving the machine benchmarking overnight and doing retrieval/needle tests too. Mostly just wanted to give the Strata dev some props. Having spent a week messing around with a similar idea, it's pretty nice to be able to download this thing and just... watch it work. If you wanna give your hardware a run for its money, it's definitely worth playing with.   submitted by   /u/ruashots [link]   [comments]