DISCLAIMER: the prefill rate on the dashboard in the video counts cached prompt tokens as if they were read again, so it reads high. The captions show the server’s real end-to-end numbers, and so does everything below. The decode numbers on screen are correct. After the last post I wanted to try something that doesn’t fit on any one machine I own. Qwen 3.8 Flash Next is a 48-layer mixture-of-experts model with 512 experts per layer, and even at 3-bit it’s far bigger than my 24 GB MacBook. But my house has more silicon than that: a gaming PC with an RTX 2060 Super (32 gigs DDR4), another with an RX 9060 XT (16 gigs DDR5), the M4 Pro MacBook, and two iPhone 17 Pro Maxes. So I chained all five into one model. How it’s split: PC 1 (2060 Super, 8 GB): layers 0–9, the output layer, and the model’s own multi-token-prediction draft head. MacBook Pro (M4 Pro, 24 GB): layers 10–29. The two iPhones (A19 Pro): for layers 10–18, each token picks 10 of 512 experts. The Mac sends those picks over USB-C, and each phone’s GPU runs them and answers in about 1 ms. The Mac keeps the most-used experts in its own memory and runs those itself. PC 2 (9060 XT, 16 GB): layers 30–47. The PCs and the Mac talk over Ethernet, the phones over 10 Gb/s USB-C. Every weight sits in VRAM, pinned RAM or unified memory, so nothing streams from disk and nothing swaps. Five devices in a row would normally mean four of them waiting. The trick is pipelined speculation. The draft head on PC 1 guesses the next few tokens, the whole chain checks them in one pass, and while that batch is still travelling through the Mac, the phones and PC 2, PC 1 is already drafting the next one. Each step lands about 2.2 tokens on average. Measured at the server with what an agent actually sends (temp 0.6, top-p 0.95, top-k 20, medium thinking, real code prompts, not greedy): 16k context: 42 tok/s writing, 728–744 tok/s reading the prompt 32k context: 39–41 tok/s writing, 844–852 tok/s reading 60k context: 37–39 tok/s writing, 856–895 tok/s reading In the agent session from the video, with the prompt cached, a short follow-up turn starts in about half a second. A turn with new tool output costs about 2.9 s fixed plus 768 tok/s for the new tokens. Writing averaged 35 tok/s across that session, since it swings with how predictable the text is. I’ve seen bursts past 40. Long context in this layout. It serves 69.6k. PC 2’s 16 GB is the ceiling. A second layout moves two layers onto PC 1 and keeps 8-bit KV on the Mac and phones. That one read a single prompt past 164k tokens, and it writes 31–33 tok/s at 60k. Full 262k needs each phone holding one attention layer’s cache, which is how the two-A19 split works. One of the A19s had to leave today, so that run is next. I believe this can go faster with more optimizing. Fast prompt reading on long prompts. PC 2 caps it at about 980 tokens/s, because its host-memory experts cross PCIe once per 512-token batch. The final quality check. Every change is exact by construction or exact in distribution, except the Mac’s expert cache, which adds partial sums in a different order. The KL check against a reference for today’s changes hasn’t run yet. Try with your phones or iPads and see how far & fast you can go! With other quants, you could definitely go much faster. Coming to the repo shortly. https://github.com/StayLameBro/backburner   submitted by   /u/StayLameBro [link]   [comments]