Disclosure: I wrote this post with the help of an AI assistant for structure and wording. All numbers come from my own runs and logs. I spent the evening tuning my llama.cpp router presets and measured everything along the way. Some findings apply beyond my setup, so here they are. Hardware / software Ryzen 7 9800X3D, 128 GB DDR5-6000 CL32 (2×64 GB), ASRock X870E Taichi RTX 5090 (PCIe 5.0 x8, also drives the desktop) + RTX 3090 (PCIe 4.0 x8) Linux (Kubuntu 26.04 LTS), llama.cpp built from source (CUDA 13.4), build 0.4.1-dev (build 11028, commit 972d2313b), router mode with a presets INI Models (Unsloth): Qwen3.8-27B UD-Q6_K and Q8_0, Qwen3.8-Flash-Next UD-Q4_K_XL flash-attn on, KV cache q8_0, single stream, MTP speculative decoding on the 27B 1. lazy-mode regression on MoE with experts in RAM Recent builds map the large embedding table lazily even with mmap disabled. On Flash-Next with experts offloaded to RAM (n-cpu-moe), this costs a lot of prefill. Same 20.6k-token document, ubatch 4096, everything else identical: Setting Prefill lazy-mode on + load-mode mmap 704 t/s lazy-mode off + load-mode mmap 1,104 t/s lazy-mode off + load-mode none 1,220–1,234 t/s With the default ubatch (512) on top of lazy-mode on + mmap, I was at ~300 t/s. During Flash prefill, nvtop shows the 5090 receiving ~28 GiB/s, which is a saturated Gen5 x8 link. Prefill is bound by expert weights crossing PCIe on every micro-batch. 2. For offloaded MoE, ubatch matters far more than n-cpu-moe ubatch 2048 → 4096 on the 32k preset: 941 → 1,234 t/s (+31%) Moving 2 more expert layers back to GPU: +3% Bigger micro-batches mean fewer weight transfers per token. Note that n-cpu-moe offloads the first layers, which live on the first device. Lowering it loads CUDA0, not the second card. To use the second card's spare VRAM, shift tensor-split toward it first. 3. batch-size = ubatch-size kills pipelining on a 2-GPU layer split 27B Q8 split across both cards, fully in VRAM, same 33k document: batch / ubatch Prefill 2048 / 512 2,669 t/s 2048 / 2048 2,040 t/s 4096 / 1024 2,702 t/s With one micro-batch per batch, the two GPUs work one after the other instead of overlapping. Keep batch at 3–4× ubatch. This is the opposite of point 2, where the bottleneck is PCIe, not inter-GPU overlap. 4. The 5090 alone beats the split for the 27B Q6 on the 5090 alone: 2,766 t/s prefill, ~100 t/s decode (33k document). Q8 split on both cards: 2,687 t/s / 69.5 t/s. Not the same quant, but the 3090 clearly paces the split. I now run Q6 on the 5090 with 128k context by default and only use the split for >128k. 5. At 262k context, the compute buffer lands on the second card Flash at 262k with ubatch 2048 failed with a 7.1 GiB compute buffer allocation on CUDA1 (the 3090). It scales with ubatch × context. Fix: keep ubatch 1024 at 262k, or use a 128k preset where ubatch 2048 fits. MTP: draft 3 is the sweet spot On the 27B, spec-draft-n-max 2 → 3: +12% decode (61.9 → 69.5 t/s). 4: +   submitted by