I've been running FreeToken on my 2x3090 box for a while and ended up maintaining a fork of it. Posting it in case it's useful to anyone else here. Quick context if you haven't used it: FreeToken is an edge-native MoE serving engine. It offloads experts to host RAM/NVMe and co-executes on CPU+GPU so you can run big MoE models on consumer hardware. Upstream is here: https://github.com/FlashML-org/FreeToken What I added on top of upstream: - DeepSeek-V4.1-Flash (mHC, CSA2 sparse attention, lightning indexer, Engram n-gram memory, DSpark draft) - Vision for Qwen3.8-Flash-Next, incl. OpenAI image_url input end to end - Speculative decoding: MTP draft head (embedded or external artifact), rejection sampling, commit-the-accepted-prefix instead of re-extending, optional round chaining - FTW (the fast weight format) now carries side tables (PLE/Engram) and the MTP head, plus TP slicing - Tensor-parallel loading for a few families, optional fp8 all-reduce - Reworked the quant config layer so each module's scheme comes from one place - Various fixes (FTW TP band offsets, O_DIRECT reads, scheduler spec budget, Engram hashing) My box, for reference: - 2x RTX 3090 (24GB each, PCIe x16), tensor parallel = 2 - AMD EPYC 7203P, 8 cores / 16 threads, 8x DDR4-2400 - Model: nvidia/Qwen3.8-Flash-Next-NVFP4 converted as FTW (512 experts, offload) - Serve flags: --moe-strategy offload --tensor-parallel-size 2 --moe-cache-auto --ple-backend pinned --expert-load parallel --max-prefill-length 8192 --disable-moe-prefill-overlap Bench (256 output tokens, 5 runs, fixed prompt, tok/s avg / median): - greedy, MTP off: 48.0 / 52.4 - sampled (temp 0.8), MTP off: 46.2 / 50.9 - greedy, MTP on: 21.5 / 22.4 (accept rate ~56%) - sampled, MTP on: 34.0 / 43.8 (MTP is gated off for sampled by default) So with CUDA graphs on the plain path, MTP is currently a net loss on this config. Under matched conditions (eager, no overlap) it was ~1.5x plain (17.2 vs 11.4), and the per-batch-size verify graph moved the MTP path from 16 to 28 tok/s in isolated runs. That verify graph is the lever I'm on right now, not claiming a win yet. Still text-only serving like upstream; vision is the one thing I wired up. DeepSeek-V4.1 keeps its Engram table on disk by default, so budget the space. Repo (anonymous fork, Apache-2.0, upstream credit kept): https://github.com/anon761/freetoken It's a fork, not upstream, so expect rough edges. If you try it and something breaks, reply here or open an issue and I'll take a look. Also curious which models/quant formats people would want next.   submitted by   /u/ApeGrower [link]   [comments]