I am working on making AI accessible by running large MoE models on affordable, consumer-grade hardware that people already own: smartphones, laptops,normal desktops. Models several times bigger than RAM, no big GPU needed. Disclosure: I’m the author of BigMoeOnEdge. All numbers below are real runs on my own or contributors devices, same engine. My vision? To act as a layer on top of llama.cpp for streaming MoE models. I’ve developed the concept of "adaptive streaming Planner" tailored specifically to the hardware, OS, CPU, GPU, and NPU in use. The algorithm assesses the environment to determine the optimal allocation of resources, with the goals of: maximizing prefill and decoding speeds, and optimizing memory management without overloading the operating system. Furthermore, the OpenAI-compatible server enables the use of local AI on consumer-grade hardware; I’ve personally used it with OpenCode - first running it on Android, and later on a Mac. What the planner alone does (16 GB Mac, Qwen3.6-35B-A3B Q4_K_M) Before With --auto (Planner Decode 7 tok/s 25-30 tok/s 2000-token prompt 21 s on CPU 3.4 s on GPU By device - but you can run any MoE gguf and quant supported by llama.cpp Model On disk Device × its RAM Decode Qwen3.6-35B-A3B 21 GB 12 GB Android phone 1.9× 10 tok/s 100-300+ tok/s prefill (NPU) Qwen3.6-35B-A3B 21 GB 16 GB Mac 1.3× 30+ tok/s 400 tok/s prefill (GPU Metal) Qwen3.6-35B-A3B 21 GB 16 GB Windows laptop 1,3x 5 tok/s Qwen3-Coder-30B-A3B 18.6 GB 16 GB Mac 1.2× 23 tok/s, 10 inside opencode Qwen3.8-Flash-Next 125B 76 GB 16 GB Mac 4.8× 6.5-9 tok/s Qwen3.8-Flash-Next 125B 80 GB 12 GB Android phone 6.7× 2.2-3.5 tok/s DeepSeek V4 Flash 284B 91 GB 12 GB Android phone 7.6× 1 tok/s I’d be happy to hear your thoughts on this and whether it might be useful. I’m considering opening a PR on llama.cpp to integrate it, but I’m not sure if that aligns with the goals of their project.   submitted by   /u/dai_app [link]   [comments]