I've been experimenting with ways to run MoE models that don't fit comfortably in RAM, and I ended up making Overspill, a disk tier for FreeToken. The basic idea came from looking at how Colibri handles experts across disk/RAM/VRAM so I took inspiration from the general approach. Repo: https://github.com/IvanAdriazola/overspill Apache-2.0 · experimental My hardware RTX 3060 12 GB Ryzen 9 7900 64 GB DDR5-6000 (WSL2 capped at 48 GB) NVMe, accessed through WSL2 Windows 11 + WSL2 Ubuntu 24.04 I tested DeepSeek-V4-Flash REAP-150B (puwaer/DeepSeek-V4-Flash-0731-reap-150b), which is ~85 GB with FP4 experts. Results Same model, same FP4 experts, cold start, greedy decoding, all on the same PC: Overspill llama.cpp Colibri Decode 2.8–3.4 tok/s 0.4–0.5 tok/s 1.1–1.2 tok/s TTFT, 6.4k prompt 102 s 373 s ~26 min TTFT, first short prompt after cold start 43 s 44 s 25 s TTFT, next short prompt 10 s 40 s 18 s These are just my measurements on this particular machine, so I wouldn't read too much into the comparisons yet, also i don't consider myself an expert, there was some heavy vibecoding invoved. The non-expert weights also aren't identical between the engines (the experts are identical in all three, but llama.cpp's GGUF stores the ~8 GB of non-expert weights (attention etc.) in Q8_0, while FreeToken and Colibri use DeepSeek's original FP8, so the runs aren't bit-identical). What I changed The main things I experimented with were: Memory-mapping experts that don't fit in RAM, letting the OS page cache act as another tier. Using madvise(WILLNEED) so Linux reads each layer's routed experts in parallel with large reads, instead of pulling them in page fault by page fault. Keeping the embedding/output layers in RAM so I could free some VRAM. Using larger prompt chunks to reduce how often the experts have to be streamed. Running short prompts on the CPU instead of moving the full expert set through the GPU. The biggest improvement I saw was expert loading from disk, which went roughly 4× faster in my tests. I also changed FreeToken's checkpoint converter, which was running out of memory on models larger than RAM. The fix worked for me, but I'd like the FreeToken devs to confirm that it's the right approach. Sanity checks On Qwen3.6-35B-A3B, which fits in RAM, I get byte-identical output to stock FreeToken on the prompts I tested. I also managed to run the DeepSeek model through a coding test and some multi-turn tool-calling tasks, although I haven't done anything resembling a comprehensive evaluation yet. One thing I tried that didn't work well was prefetching the next layer's experts from RAM → GPU. It was functional, but ended up 9–29% slower on my 3060. My current guess is that the transfers are competing with GPU computation, but I could be misunderstanding what's actually happening. This is very much an experimental proof of concept right now: one machine, one large model, WSL2, and one request at a time. In Overspill's disk path the expert math runs on the CPU (FreeToken's CPU executor, which used the AVX-512 path on my Zen 4 Ryzen 9 7900), and the experts stream from disk through RAM. So these numbers depend heavily on the CPU, RAM speed (DDR5-6000 here) and storage, not just the GPU. A CPU without AVX-512 (many Intel consumer chips) falls back to slower code paths, and fewer cores, slower RAM, a slower SSD or less RAM for the page cache will all likely lower decode speed. Please don't read my ~3 tok/s as a general figure; treat it as what one fairly strong CPU + DDR5 + NVMe setup gets, and I'd really like to see how it scales on other machines. If anyone with native Linux, faster storage, more RAM, or different hardware wants to try it, I'd be very interested in the results. And if I've misunderstood something about FreeToken, Colibri, mmap/page caching, or the performance measurements, please tell me. Most of this is thanks to the existing work from the FreeToken team and the ideas Colibri came up with. I mainly put a relatively small experimental layer on top of FreeToken to see whether this approach could work with disk-backed experts. Final warning: the LLM space moves ridiculously fast, so there's a good chance this is already old news by the time I post it. Sorry in advance if someone already got this. 😅   submitted by   /u/Chekhovs_Shotgun [link]   [comments]