hey folks, im Ye, im one of the contributors to Kapitsa, an open source inference engine optimized specifically for Apple Silicon (all the way from M1 chips to M6) to run Qwen & Nemotron models that would fit a regular MacBook. its custom written Metal kernels, allow us to optimize each chip & for each model. https://github.com/nace-ai/kapitsa/ try it (needs a Mac with 32 GB or more; the download is 22.6 GB): brew install nace-ai/tap/kapitsa kapitsa serve --model flint # OpenAI and Anthropic APIs on localhost kapitsa opencode --model flint # or claude, codex, pi kapitsa bench recursive released a model today and they claim new model that is optimized for coding on 32gb+ macs, its Flint, they claim to beat 35b sparse MoEs by Qwen and come close to qwen 27b dense. we went ahead and added support of Flint to Kapitsa, reaching SOTA decode/prefill on sub M5 chips. Speculative Decoding: we also span up 8-H200 rack to train a tree drafter for KFlash speculative decoing specifically for Flint Q4 to run faster on any M chip, which internally gives us 1.4x speedup on current numbers already for decode without any acuracy loss, hopefully pushing it to ~1.5x. to be released on Monday. by our metrics a KFlash tree drafter trained on q4 flint, really performs well on the fp16 too. here are the benchmarks: Recursive Benchmark of Flint Prompt Reading Speed Writing Speed Setup. MacBook Pro 16 (2024), M4 Pro, 48 GB, macOS 27.0.1, plugged in, High Power mode. The model card's sampling: T 1.0, top_p 0.95, top_k 20. One engine at a time on the GPU. No speculative decoding anywhere: Flint has no draft model yet. Kapitsa rec 0.1.0 llama.cpp b11804 Decode, short context (tok/s) 127.9 126.6 79.6 Prefill, 4K-token prompt (tok/s) 968 796 977 Prefill, 16K-token prompt (tok/s) 817 699 786 Decode at 128K context (tok/s, one request) 45.7 35.2 23.8 Prefill at 128K (tok/s, one request) 300 262 199 Follow-up turn, time to first token (s) 0.13 0.41 0.23 Peak memory (GB) 23.3 27.9 26.9 What the numbers say: Short context: decode is a tie with rec(recursive's own engine). The two are 1% apart, and greedy, rec is 0.6% ahead. Both decode 1.6× as fast as llama.cpp. Long context: Kapitsa pulls ahead. Its decode lead over rec grows with the context, from 1.06× at 8K to 1.30× at 128K (the chart has every point). Against llama.cpp it is 1.6× up to 64K and 1.9× at 128K. Prefill: 1.1-1.2× rec at every length. llama.cpp reads short prompts a little faster (1-8% up to 4K) and is level at 8K. From 16K on, Kapitsa is ahead, by 1.5× at 128K. Follow-up turns start 3× sooner than on rec: 0.13 against 0.41 s to the first token of a cached second turn. the sweep scripts are in the repo: bench/results/2026-10-22-flint-m4pro. things to mention Flint always reasons before it answers, so its reasoning tokens count toward decode on every engine. one Mac. We haven't run Flint on an M5 yet. new code, so expect bugs. prefill on M5+ is slow and has not been optimized yet, WIP. our CTO will get us a mbp m5 max 128gb to optimize on, if we pass 1k stars lol, help asks we have a command called: kapitsa bench it will publish a PR with your benchmarks of the models that we do not yet have, for flint we still lack a wide variety of devices, so please feel free to contribute and help us around with the benchmarks. We still need M5 Max, M5 Pro, M5 ultra devices to check the speedup for Flint. kapitsa 0.2.7 bench happy to answer questions and take advices on where to steer the project.   submitted by   /u/DotaMate [link]   [comments]