The problem I'm solving: I have an M3 Ultra 512 GB and the memory is still the reason to own it, but prefill on long prompts is slow. A 17.5k-token prompt takes 41 s before the first token. The M5 Ultra fixes prefill but ships with 256 GB max right now, and an M5 Ultra 512 GB isn't a thing yet. So rather than replace the M3, I added an M5 Ultra 256 GB and split the work: the M5 does prefill, the cache state goes over Thunderbolt 5 RDMA, and the M3 does decode against its 512 GB pool. Single-node numbers first, GLM-5.3-Flash 4-bit with MTP, 17.5k-token prompt, 256 output tokens. metric M3 Ultra 512 M5 Ultra 256 prefill, 2k-token chunks 438 tok/s (40 s) 1,046 tok/s (16.8 s) prefill, 8k-token chunks 454 tok/s 1,188 tok/s decode 32.9 tok/s 47.2 tok/s The M5 is about 2.4x on prefill but only about 1.4x on decode, and decode is the phase that benefits from the bigger memory (more context, more concurrent conversations). That asymmetry is the whole argument for splitting. Why "only" 2.4x when the M5's matrix units are roughly 5x? I profiled it. Raw matmul in MLX: 118 vs 24 TFLOPS bf16, 108 vs 24 on 4-bit (the quantized kernels dequantize to bf16 and run on the same units; 8-bit is no faster). But a GLM-5.3-Flash prefill is only about half matmul. Per-component, 8,192 tokens in 2,048-token chunks: component M3 Ultra M5 Ultra speedup share on M5 MoE expert matmuls (42 layers) 8.1 s 2.65 s 3.1x 34% DSA sparse-attention core (11 layers) 3.8 s 2.44 s 1.55x 31% KDA qkv projection + conv 2.9 s 0.89 s 3.2x 11% MoE gate / routing 0.77 s 0.60 s 1.3x 8% KDA linear-attention scan 0.88 s 0.49 s 1.8x 6% dense MLP + other projections 2.1 s 0.70 s 3.0x 9% DSA indexer 0.43 s 0.25 s 1.7x 3% total 18.7 s 7.8 s 2.4x Everything that is a real matmul gets 3-4x. Everything that isn't (the per-query top-2048 gather kernel for sparse attention, the recurrent scan, expert routing) gets 1.3-1.8x, i.e. roughly the memory-bandwidth ratio (558 vs 382 GB/s streaming). The M5 spends almost half of prefill in kernels that can't use its matrix units at all. Two levers fell out of this. Bigger prefill chunks: at 8,192-token chunks the M5 goes to 1,188 tok/s (+14%, the MoE expert matmuls get enough tokens per expert to stop being bandwidth-bound) while the M3 gains 4%. And the sparse-attention kernel: it handles one query per threadgroup and gathers that query's 2,048 keys with scalar dot products, so a block-sparse version that shares gathered keys across neighbouring queries and uses the matrix units is where the next big M5 gain is. The linear-attention scan everyone worries about is only 6%. Disaggregated result, same prompt: prefill on the M5: 988 tok/s, 17.8 s cache hand-off to the M3: 358 MB over both TB5 cables in 38 ms end to end first token on the M3: 38 ms after the cache arrives, so about 18.5 s total instead of about 40 s decode on the M3: 36.8 tok/s, output identical to the M5's own the M5 is free again 18 s after the request and can prefill the next prompt while the M3 decodes Why the hand-off is cheap for this model: GLM-5.3-Flash has 34 linear-attention (KDA) layers with a fixed-size state and only 11 sparse-attention layers with a 512-wide latent KV, about 20 KB per token in total. At that rate the M3's spare 300 GB or so holds on the order of 15 million cached tokens of conversations. Stack: MLX's JACCL backend (RDMA over Thunderbolt, two cables on separate controllers, about 150 Gb/s combined), mlx-vlm 0.7.6 for the model and MTP drafter, a Python prototype that serializes the cache objects into three dtype-packed arrays and ships them with mx.distributed.send/recv. The limits: The prefill node needs the full model too, so this only works for models that fit in 256 GB. Anything bigger still needs the 512 alone, or a real 512 GB M5. For one request at a time the M5 alone decodes faster (47 vs 37 tok/s). The win is time to first token on long prompts, throughput with requests in flight, and keeping the 512 GB pool in play. Things that cost me time, in case they save you some: Apple's Thunderbolt RDMA only offers unreliable-connected queue pairs, but a send simply blocks until the receiver posts a buffer. No flow control needed. Upstream JACCL never checks completion status; a sender/receiver size mismatch turns into a SIGBUS in a memmove. A Mac GPU idle for about 17 s takes about 1.25 s on its first op afterwards. Looked exactly like a flaky link until it wasn't. A keep-alive thread fixes it. mx.set_wired_limit to the recommended working set is mandatory or the model pages and decode drops to 9-17 tok/s. Is anyone else pairing Macs this way, or working on prefill/decode splits in exo or mlx-lm for this model family? I'd rather compare notes than duplicate work.   submitted by   /u/Usual_Tackle5892 [link]   [comments]