I wanted to run a larger local model on my Mac mini without having to close all other apps. With Qwen 3.8 Flash Next, I’m now getting around 10 tok/s while keeping Safari and other apps open. The runtime I’ve been working on is called MoEShift. I’ve released the alpha as open source: GitHub, installation and benchmarks. These results are from my M4 Pro with 24GB unified memory, using ISTA-DASLab’s GSQ-RCO IQ3_XXS quant (76GB on disk). I compared it with Strata 0.2.7 using the same model in cold runs. With a 6k token prompt and 256 output tokens, total inference took 63.3s versus Strata’s 129.9s. Prompt processing reached 160.8 tok/s versus 61.8, with generation at 9.9 tok/s and peak memory at 7.5GB versus 14GB. With a short prompt, generation was 10.44 tok/s versus Strata’s 8.4, with peak memory at 6.4GB versus 14GB. More details in the README. MoEShift loads experts from SSD as needed, caches frequently used experts in RAM and overlaps reads with GPU computation. It builds on llama.cpp and Lookahead Expert Scheduling. My changes focus on Qwen 3.8 Flash Next support, execution and memory reuse between prompt processing and generation. The original quantized weights and expert routing are preserved and there is no speculative decoding. The architecture doc goes into the details. I’m curious how far MoEShift can go on other Macs. If you give it a try, post your results here.   submitted by   /u/dave-tro [link]   [comments]