Hi! I'm the author of MoEspresso, which is my way of putting my own ideas about inference engines to the test. A lot of the fun has been trying different design choices, measuring what happens, and finding that several of them work well together. MoEspresso 3 runs Qwen3.8-Flash-Next on a 2021 M1 Max with 32 GB of unified memory at 12-15 decode tokens per second - provided there are no other memory hungry applications running in the background (such as browsers). During decode, experts which are already resident in memory receive a bias, but the two strongest experts according to the model are always chosen (with the default settings). I wrote about this here https://github.com/steadfastgaze/MoEspresso/blob/main/docs/cache_prior.md - I first started thinking about this after reading about Apple's AFM 3 and the instruction-following pruning work behind it (https://arxiv.org/html/2501.02086v3#abstract), but then I found this other paper (https://arxiv.org/html/2412.00099v2) which spoke about Cache-Prior. Prefill is unbiased. Even with this bias enabled by default, Qwen 3.8 Next scored ahead of Opus 4.8 xhigh and many other strong hosted solutions. Reproducible 48-question setup -> https://github.com/steadfastgaze/MoEspresso/tree/main/docs/benchmark_reproduction. Overall scores (%) across six categories, including coding, data analysis and math: Model Score Qwen3.8 Flash (hosted), medium 89.7 GPT-6 Sol, medium 89.2 Claude Opus 5.5, medium 86.4 GPT-6 Sol, low 85.0 Qwen3.8 Flash @ MoEspresso, medium, Cache-Prior 2/2 84.3 GPT-6 Luna, xhigh 81.4 Claude Opus 4.8, xhigh 80.3 Claude Sonnet 4.6, high 74.0 Claude Sonnet 5, medium 70.9 I use some of Iwan Kawrakow's formats from ik_llama.cpp, with Metal execution through my mlx-iqk library. Most routed projections in this package use IQ2_K, which is not normally supported by either standard MLX or mainline llama.cpp. KVarN K4/V4 leaves more memory for resident experts as context grows, and it is functioning extremely well with low RMS error on this model architecture. Good defaults, e.g. automatic SSD streaming and Cache-Prior when all experts cannot fit, with the settings generally following the same rule. This is the third iteration, and I have more concrete ideas to explore, both for squeezing even more performance from Apple Silicon and for bringing the engine to Linux and AMD machines such as Strix Halo. Installation is through Homebrew, so "brew install steadfastgaze/tap/moespresso" Code - https://github.com/steadfastgaze/MoEspresso Model - https://huggingface.co/steadfastgaze/Qwen3.8-Flash-Next-MoEspressoV3 If you try it, I'd love to see your "moespresso speed" output (an intentionally quick benchmark). PS: English isn't my first language and I used an LLM to help refine this post, and AI coding tools for implementation.   submitted by   /u/marcobaldo [link]   [comments]