This is a follow-up to my post from two weeks ago. The main request in that thread was to run oMLX with an oQ4e-mtp checkpoint and Lightning MTP instead of the plain mlx-community build. I did that, added Kapitsa 0.2.12 as a fifth engine, and tested Flint 40B-A2.5B. The short version: oMLX's dense number changes completely, the MoE race gets tighter, and Rapid-MLX is no longer my default. TL;DR (M3 Max 36 GB, sustained 256-token prose, median of 3): Dense Qwen3.8-27B-4bit: Kapitsa 41.3 tok/s, oMLX 34.1, Splash 33.0, MTPLX 31.6, Rapid 26.2. MoE Qwen3.6-35B-A3B: oMLX 122.9, Kapitsa 122.7 (tie), Rapid 110.3, Splash 97.0, MTPLX 79.9. Flint 40B-A2.5B: Kapitsa 134.2, Rapid 103.0. Runs on 36 GB; not my default. oMLX's dense jump from 17.9 to 34.1 is mostly the checkpoint/speculation change (oQ4e-mtp + Lightning MTP), not a free engine-only win. Draft acceptance was 75–78% dense, 81–85% MoE. My daily default is now Kapitsa 0.2.12 on the 35B-A3B (I kept the old qwen3.6-35b-4bit API alias so clients didn't change). Setup and protocol 14" M3 Max, 14-core CPU / 30-core GPU / 36 GB / 1 TB, macOS 27, AC power, High Power Mode, 30 GB GPU wired limit. Still loud under sustained load. Engines oMLX 0.7.2 (Jundot oQ4e-mtp + Lightning MTP) Rapid-MLX 0.16.8 (its MTP builds) Splash 1.4.0 (mlx-community + DFlash2) MTPLX 2.12.2 (Optimized-Speed packs) Kapitsa 0.2.12 (catalog 4-bit + MTP head + DFlash2 drafter) Kapitsa is new to me: an Apple-silicon engine aimed at coding agents, with Qwen and Flint support. One cell per engine × family, plus two Flint cells. Each cell: stop server, clear that engine's persistent KV cache, start fresh, sustained test (same short prose prompt, 3 × 256 tokens, temp 0, reasoning_effort: none where supported), 150 s gap, then a prompt-size matrix (128:3, 2048:2, 8192:1 at 256 tokens), stop. 300 s between cells. 12 cells, all completed. The weight files are not identical. Each engine runs its own best build. Same base model families, different finishing; that is the real-world choice, but it means the table is engine+build, not engine alone. I flag where that matters. Sustained decode (median of 3, cap 256, all runs hit the cap) Engine Dense Qwen3.8-27B-4bit MoE Qwen3.6-35B-A3B Kapitsa 0.2.12 41.3 tok/s 122.7 tok/s oMLX 0.7.2 (Lightning MTP) 34.1 122.9 Splash 1.4.0 33.0 97.0 MTPLX 2.12.2 31.6 79.9 Rapid-MLX 0.16.8 26.2 110.3 What changed since the last post oMLX: 17.9 / 104.2 → 34.1 / 122.9. The MoE gain is +18%; the dense gain is mostly speculation finally being used. Different checkpoint from last time, so this is not an engine-only delta. For the "oMLX is falling behind" comment: with the MTP build it is now the MoE leader and second on dense. The plain 4-bit result was misleading. Splash: 31.7 / 77.0 → 33.0 / 97.0. 1.4.0's adaptive MoE draft verification is the big change. Rapid: 27.7 / 110.2 → 26.2 / 110.3. Flat MoE, ~5% slower dense. The 0.15.7 pass in between was 27.5 / 115.2, so 0.16.8 is a small single-stream step back. Its release notes focus on Flint and concurrency, not single-stream speed. MTPLX: 31.5 / 79.0 → 31.6 / 79.9 on the same 2.12.2. That is my sanity check that the protocol is stable. The 35B greedy/MTP-depth work its author mentioned last thread is not in a release I can test yet; 2.12.2 is still latest as of this run. Kapitsa: new. 41.3 / 122.7. Ties oMLX on MoE, wins dense. The oMLX-is-20%-slower-than-Splash comment from last time is reversed here (34.1 vs 33.0 dense, 122.9 vs 97.0 MoE), but again, that is oQ4e-mtp + MTP vs Splash's DFlash2 build, so read it as engine+build. Long-prompt matrix (decode tok/s; target sizes 128/2048/8192, actual prompt tokens ~100/1.4K/5.5K) Engine ~100 tok ~1.4K tok ~5.5K tok Kapitsa dense 56.1 52.7 48.2 Kapitsa MoE 306.0* 226.5* 224.1* Kapitsa Flint 133.8 129.3 119.3 oMLX dense 37.5 35.2 35.3 oMLX MoE 146.6 131.1 114.3 Splash dense 50.4 49.9 71.1* Splash MoE 273.9* 211.3* 218.1* Rapid dense 29.4 27.8 24.5* Rapid MoE 213.2 122.6* 114.6* Rapid Flint 102.5 96.5 91.8 MTPLX dense 36.3 30.0 26.0 MTPLX MoE 81.3 78.9 70.5 * = at least one run stopped before 256 tokens (36–214). Those rates are short-sample bursts, not comparable to the sustained table. Repetitive filler also makes speculative decoding look better; the prose table is the fair one. 8K TTFT (time to first token): Kapitsa 3.8 s MoE / 32.9 s dense; Splash 4.1 / 22.0; Rapid 4.4 / 35.1; oMLX 4.5 / 35.7; MTPLX 4.9 / 36.7. Splash's Neural Engine prefill is still the fastest way to read a long dense prompt, even though its sustained decode is third. TTFT is valid even where the decode run short-stopped. Standardized rapidmlx protocol (pp512-tg128 / pp2048-tg512, 1 warmup + 5 measured) Model pp512-tg128 pp2048-tg512 TTFT 512 / 2048 qwen3.5-9b-4bit 54.1 53.6 1.0 / 4.7 s qwen3.8-27b-4bit 17.5 17.6 3.3 / 12.8 s qwen3.5-4b-4bit 90.7 90.7 0.6 / 2.2 s gemma-4-e4b-4bit 73.5 73.6 0.3 / 1.3 s bonsai-1.7b-2bit 272.8 228.9 0.2 / 0.7 s qwen3.6-35b-4bit 102.2 100.1 0.5 / 1.6 s The 27B row is the MLLM/vision lane (17.5), not the serve text lane (34.1 with MTP); that is a lane difference, not a contradiction. All six completed, including qwen3.6-35b-4bit, which used to abort at 21 tokens. I shared the six runs on the leaderboard. Flint 40B-A2.5B Kapitsa 0.2.12 --model flint, RecursiveSI/flint-int4 4-bit MLX, 22.6 GB. Loads in ~9 s, 253,952 context, 4 slots, plain decoding (no draft/MTP). 134.2 tok/s sustained; 119.3 at 8K. Rapid 0.16.8 flint-40b-a2.5b-4bit, same checkpoint. Runs on 36 GB but 48 GB+ is recommended; no speculation. 103.0 tok/s; 91.8 at 8K. oMLX 0.7.2, Splash 1.4.0, MTPLX 2.12.2 I could not find Flint support. Not my default model 36 GB is tight, and the 35B-A3B is the better daily agent model. But a 40B at 134 tok/s on a 36 GB laptop is the most surprising result of this pass. Quality caveats No formal quality eval. This is a speed post, not a quality verdict. That was the main gap people pointed out last time, and it is still not closed. Speculative decoding can change outputs near ties. oMLX MTP acceptance was 75–78% dense / 81–85% MoE. Acceptance is not quality, but it tells you how much of the speed is speculation. Splash 1.4.0's own log warns that MoE --verify-rows auto can change greedy/seeded outputs run to run. If you need deterministic outputs, use --verify-rows all and/or turn speculation off. I did not soak-test memory for 12–24 h. The longest cell here is ~5 minutes. The oMLX memory complaint from the last thread is not tested by this pass; I can only say I did not hit a problem in these short runs. One laptop, one session, no die-temperature logging. The 150 s/300 s cooldowns are the thermal control. Not tested / will do later The unsloth GGUF suggestion from the last thread. A proper KL/task quality eval next to the speed table. Long agent sessions (tool calls, multi-turn) rather than single completions. A 12–24 h memory soak, especially for oMLX. What I'd pick 35B-A3B daily Kapitsa 0.2.12. It ties oMLX on MoE (122.7 vs 122.9, inside noise), wins dense, and documents JSON-schema structured output; oMLX's brew build still lacks xgrammar. Rapid is still a reasonable conservative choice if its agent integration is what you rely on, but it is ~10% behind on the served MoE workload now. Dense 27B Kapitsa 41.3, then oMLX 34.1. If long-prompt prefill matters more than decode, Splash's 8K TTFT (22 s vs ~33–37 s) is worth a look. 40B: Flint on Kapitsa. Final note: No affiliation with any of these projects. I'm just a user with a Macbook.   submitted by   /u/Hyungsun [link]   [comments]