Been talking about releasing my streaming inference stack for a while so here you go all. It's a bit of a medley of things. TLDR: Run dsv4.1 at 1700tok/s prefill and 7-20tok/s decode full precision weights variable speed/quality tradeoff. Can also prefill on rtx machine and decode on a mac or big ram server. Directstorage streaming prefill from fast nvme of models WAY too big to fit. Then either decode locally with router biasing to prefer weights in the vram cache for 20tok/s dsv4.1 or 7 tok/s just full precision normal decoding with most of the experts streaming with directstorage from nvme. Other alternative is after the streaming prefill you pass the kv/state over network to another machine, big unified moemory mac or a server with lots of ddr but not much compute. decode on the big memory machine. Webui has image and video streaming as well so you can run models that dont fit in your ram and vram. I have run full precision minimax h3 and magi 2 preview off the one 5090 without touching RAM and its no slower than resident if you have a decent ssd and the video is big enough compute takes longer than weight streaming because they overlap. There is some experimental lora streaming/training/merging in there along with the recipe to train a simple identity lora for dsv4.1 that makes it answer any question about its identify by calling itself jeff. Trained again on a single 5090 in about 20 minutes. Webui also lets you set the router bias to be dynamically changeable so you can pull the slider to max quality for 7tok/s full precision without compromise or to max speed for 20 tok/s with a kl in the region of IQ1 level KL or anywhere in between. You can also set phases so it reasons fast but writes code blocks at max quality and non code answers at a balanced level or whatever. This is all using the full unquantised dsv4.1 recipe Q4/Q8 and the script to make the gguf I used from the safetensors is in there as well. There is example scripts to run it and you can run different platforms on each machine so you can run streaming llama on prefill end and mlx or mlxserve or omlx or whatever for decode on a mac so you can mix and match engines for whatever is faster and it does the kv translation. And that's about it. Problems: Minimum answer latency single machine is however long it takes to stream the whole model at your SSD/raid arrays best speed. My dual pcie5 array does 25GB/s so gets through a dsv4.1 prefill in about 20 seconds. Dual machine uses the decode machine for prefill if it will be slower by deciding if the prompt is above or below the crossover token count. Basically you get mac prefill speed for 2000 token prompts, rtx for bigger. Streaming decode support is limited right now just dsv4.1. glm5.3 flash is in the works. Streaming prefill supports glm 5 series, step 3 models, dsv4.1, kimi k2.5/6/7 and maybe one or two others I have forgotten about. No linux support yet. I need this machine running windows for supporting other things and the raid array wont support linux (I have tried) so I had to pick one. Hoping to get a dedicated linux box when funds allow. RTX 50 series/rtx6000 series/blackwell cards only so far. Easy enough to add non blackwell support but I dont have one for testing. Let me know what you think. Repo: https://github.com/fronteagle739/streaming-llama-stack   submitted by   /u/Front_Eagle739 [link]   [comments]