Second and last post on this because I have more than doubled performance from last time. Full fp4/fp8 dsv4.1 streamed from raid array for >4000 tok/s prefill for long prompts then drops the prefill buffers and loads a hot set of experts into a cache and decodes with heavily optimised directstorage fetches for missed experts. gets 7.5tok/s decode with no divergence from any other engine but you can bias away from nvme resident experts with a slider to trade speed vs kl divergence per token. Roughly 1.5bpw equivalent KL divergence gives me 24 tok/s and there is a slider you can adjust at will when you want speed or quality or set the webui to switch the bias for fast reasoning, quality code blocks/tool calls and balanced non tool call answers or whatever else you want. Has some other bits as well, can send kv to mac and decode on that. Also has image support and video support on the same streaming system so you can chat in the webui, request images or videos and they just appear in the same chat as it switches between them. Doesn't use your ram for image or video model streaming. If you have more ram its worth turning on the ram cache and cpu expert processing but at 32GB I cant fit enough to be useful. (faster your nvme the better the whole thing works, I have a dual pcie5 512GB raid0 array). Feedback from anyone that tries it would be nice as its not been tested on any machine but mine. https://github.com/fronteagle739/streaming-llama-stack   submitted by   /u/Front_Eagle739 [link]   [comments]