What I built and tested, on identical prompts (20k input) with greedy output for a fair comparison: │ Build │ Generation (tok/s) │ Verdict │ │ Your installed v0.7.0 (control) │ 168.8 │ best, together with the settings below │ │ v0.7.0 + smarter expert-cache eviction (#30637) │ 171.5 │ within noise │ │ Latest upstream master (30 more commits) │ 169.6 │ within noise │ │ Master + #30637 │ 168.9 │ within noise │ │ Strata-style cache pre-filled from an expert profile │ 165.8 │ only helps the first seconds │ │ Strata-style hybrid (CPU computes cache misses) │ 89–98 │ much slower; 24 threads didn't help │ Strata and Magnitude: - Strata: its one transferable piece is the expert-routing profiler. It works with llama.cpp through #30594, so I built the tracer and made a profile for Bolt. The hybrid mode loses here because your GPU link (PCIe Gen5 x16) uploads a missing expert faster than the CPU computes it; it's meant for laptops with slow PCIe. The profile alone barely matters, because the GPU cache already hits about 82% after warming up. - Magnitude: it doesn't support Bolt and shows no support for models bigger than VRAM, so nothing to take. Settings found this round: - Larger prompt chunks: chunk 2048 reads prompts at about 1,960 tokens/s instead of 1,600 but generates about 10 tokens/s slower. 4096 reaches 2,178 reading with generation down to 142. - Expert cache size: 7.5 GB is the safe maximum. 7.8 GB starts fine but runs out of GPU memory mid-generation. - Draft settings: the defaults are best. 3 tokens per step, and no confidence threshold (0.3 gave 164, 0.6 gave 156).   submitted by   /u/yehyakar [link]   [comments]