hey there, I measured Nanbeige4.3-33B (dense, 4-bit) on three Apple silicon engines: mlx-lm, llama.cpp and Kapitsa. The comparison covers prefill, decode, time to first token, follow-up latency and memory, at prompt lengths from 512 to 32K tokens. its a new model, folks from nanbeige claim these numbers. https://preview.redd.it/328gzbcvvvxh1.png?width=2800&format=png&auto=webp&s=d296f9e03f66c98ab4f5da70e31d5a5be8c3c336 things to mention: I work on Apple Silicon Inference engine: Kapitsa, its open source apache2, so treat the numbers with that in mind. The raw outputs and scripts are linked at the bottom so anyone can check or rerun them. Setup Machine: MacBook Pro M5 Max, 128 GB, macOS 26.4, AC power, High Power mode. Weights: mlx-lm 0.32.2 and Kapitsa 0.2.19 read the same 4-bit MLX AWQ pack (group 64). The current release, 0.2.20, adds two KFlash fixes for long contexts and changes none of the numbers here. llama.cpp b11871 reads Q4_K_M. Method: the engines ran one at a time, in turn, under the same conditions. TL;DR This model is SLOW, but very inefficent KV-cache, every layer having full attention isnt optimal i guess, muse glimmer is 1.6gb for 128k tokens but nanbeige is 26gb lol. Best Prefill mlx-lm Decode Kapitsa Time to first token, cold prompt mlx-lm Time to first token, follow-up turn Kapitsa Peak memory at 29.5K context Kapitsa 1. Prefill speed prefill Context mlx-lm llama.cpp Kapitsa 512 757 691 693 1K 829 669 721 2K 840 639 731 4K 773 556 692 8K 663 496 616 16K 618 437 540 32K 546 349 520 Prompt tokens per second, from each engine's own benchmark tool. 2. Decode speed vs context length Context mlx-lm llama.cpp Kapitsa with KFlash 512 28.8 27.3 34.9 1K 28.4 26.9 32.6 2K 27.9 26.4 31.0 4K 27.5 24.6 30.3 8K 25.8 23.2 26.1 16K 24.3 20.4 25.6 32K 20.8 16.7 22.8 https://preview.redd.it/783i1abbxvxh1.png?width=1760&format=png&auto=webp&s=d60f9f66adc92a0e0f04b4394845ac617f5033c8 3. Time to first token, cold prompt ttft cold Prompt tokens mlx-lm llama.cpp Kapitsa 532 0.90 s 1.20 s 1.18 s 995 1.75 s 1.94 s 1.94 s 2K 3.36 s 4.28 s 4.07 s 4.1K 6.44 s 8.71 s 8.08 s 8.1K 12.9 s 17.9 s 15.1 s 16.1K 28.0 s 40.7 s 33.2 s 29.5K 59.4 s 89.7 s 69.4 s These are real chat requests to each engine's OpenAI-compatible server, with thinking off, so the times include the chat template and the server's own overhead. The order matches the prefill results. https://preview.redd.it/053ey6ihxvxh1.png?width=1760&format=png&auto=webp&s=8003339f13619c909071dc6ac9efe1205098f6a0 4. Time to first token on a follow-up turn ttft follow-up Conversation so far mlx-lm llama.cpp Kapitsa 532 290 ms 220 ms 140 ms 995 310 ms 220 ms 150 ms 2K 340 ms 220 ms 160 ms 4.1K 370 ms 240 ms 150 ms 8.1K 460 ms 270 ms 190 ms 16.1K 580 ms 320 ms 230 ms 29.5K 880 ms 410 ms 300 ms The conversation so far is already cached, and a new user turn is added. This is the latency you feel in multi-turn chat and agent loops. All three engines reuse their cache here; they differ in how much each turn costs on top of it. https://preview.redd.it/04ilir5fxvxh1.png?width=1760&format=png&auto=webp&s=1002135edfc3817bbb4f70cb4b38a0c2f8cc3d48 5. Peak memory as the context grows memory Context reached mlx-lm llama.cpp Kapitsa 532 21.9 GB 28.0 GB 22.6 GB 2K 22.3 GB 28.6 GB 22.6 GB 4.1K 24.9 GB 29.1 GB 22.8 GB 8.1K 31.6 GB 30.2 GB 24.4 GB 16.1K 36.7 GB 32.0 GB 25.4 GB 29.5K 52.1 GB 35.5 GB 28.1 GB https://preview.redd.it/4eozm8faxvxh1.png?width=1760&format=png&auto=webp&s=421e6612178ee93e86e75db87b5279f383422c29 notice the growth of memory with the context growth 6. Writing code with speculative decoding writing code mlx-lm llama.cpp Kapitsa (KFlash) greedy (T 0) 27.1 tok/s 24.9 tok/s sampled (T 0.6) 26.7 tok/s 24.8 tok/s The task was a 512-token Python function with unit tests, sent through each engine's server. KFlash is Kapitsa's speculative decoding, which kapitsa pull nanbeige sets up. mlx-lm and llama.cpp run plain decoding here, because neither has a speculative decoding setup for this model that I know of. If you know one, tell me and I'll add it. Greedy output is token-for-token identical with KFlash on and off. The prompt here is short. Past about 2K tokens of context, KFlash mostly switches itself off, and decode runs within about 5% of the speeds in section 2. How Kflash works Kflash - speculative decoding, but as a tree. a tiny drafter model guesses the next 7 tokens and the 27b checks all of them in one pass. most engines check those guesses as a chain, so the first wrong guess throws away everything after it. kflash (our version) checks a tree instead: where the drafter isnt sure, its second guess comes along in the same pass, and we keep the longest branch the 27b agrees with. heres one pass both ways: https://preview.redd.it/7ajhl1d2vvxh1.png?width=880&format=png&auto=webp&s=9305f1bb5e32e226272fa11d33073206300fc2a3 One cycle, two ways to spend the same 8 rows (an illustration). Top: the drafter's chain. The target accepts key, then writes not, and the rest of the chain is discarded. Bottom: a best-first tree of the same candidates. Rows are numbered in the order the builder takes them, with each node's path probability. The target accepts key, not and in, then writes self. Positions count from P, the last token written. To reproduce the Kapitsa side: brew install nace-ai/tap/kapitsa kapitsa pull nanbeige kapitsa serve --model nanbeige   submitted by   /u/DotaMate [link]   [comments]