I am not complaining, I just want to understand. So I am running llama-server on a headless Linux with Qwen3.6 35B A3B at a Q5_K_XL quant (26GB safetensor) and 132k context and KV cache at q8_0. I have a 4060TI 16GB with an old i7 gen 7 processor and 48GB DDR4 sysRAM. When I run a dense model of like 26B (Gemma or Qwen) at a Q4_K_M quant I push it with -ngl 25 or so to the limit of my available VRAM and then creep at around 2 t/s through molasses. But with the MOE version I get relatively good speed while only occupying 4.5GB VRAM: > $ ./build/bin/llama-server -m /mnt/nvmepci/llms/unsloth/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf \ -c 131072 \ -np 1 \ -ngl 99 \ --cpu-moe \ -fa on \ -ctk q8_0 \ -ctv q8_0 \ --host 0.0.0.0 --port 8000 22.18.899.555 I slot print_timing: id 0 | task 343 | n_gen = 218, tg = 19.54 t/s, tg_3s = 19.57 t/s 22.21.923.659 I slot print_timing: id 0 | task 343 | n_gen = 277, tg = 19.53 t/s, tg_3s = 19.51 t/s 22.23.102.888 I slot print_timing: id 0 | task 343 | prompt eval time = 13539.35 ms / 1795 tokens ( 7.54 ms per token, 132.58 tokens per second) 22.23.102.892 I slot print_timing: id 0 | task 343 | eval time = 15308.76 ms / 300 tokens ( 51.20 ms per token, 19.53 tokens per second) 22.23.102.892 I slot print_timing: id 0 | task 343 | total time = 28848.11 ms / 2095 tokens 22.23.102.893 I slot print_timing: id 0 | task 343 | graphs reused = 621 22.23.103.045 I slot release: id 0 | task 343 | stop processing: n_tokens = 2094, truncated = 0 22.45.681.142 I slot get_availabl: id 0 | task -1 | selected slot by LCP similarity, f_sim_best = 1.000 (> 0.100 thold), f_keep = 0.857 That's almost 10 times as fast as with the dense model and it leaves me puzzled. What's the catch here? I know it caches a lot in sysRAM however shouldn't that be an absolute bottleneck to push expert layers for each token through the PCIe lanes with that relatively old CPU and slow DDR4?   submitted by   /u/dreamyrhodes [link]   [comments]