Dual A100X PCIe (link at Gen4 x8, weird DPU cards) in a 2U EPYC 7402P box, 256GB DDR4. Proxmox with the GPU's passed through to debian 13, 610.43.02 drivers. llama.cpp, qwen3.8 flash next UD-Q4_K_XL, Q8_0 MTP head, draft max 2, mmproj enabled. -sm layer -fa on --parallel 7 --ctx-size 1835008 -ub 2048 --cache-ram 32768 (7 slots x 262k), q8_0 K/V Real world single user performance sits around 50-70tk/s. When bumping things up to 2-3 users/sessions, generation drops like a rock, 10-15tk/s. Seeking advice on how to better serve my team at work. We need models worth running and performance that doesn't have us waiting around. I'm very partial towards MoE models, so no 27B. Should we swap over to vLLM, or run an instance of qwen3.6-35b-a3b on each card? Use case is long session agentic 'coding' (more running linux commands, probing hardware devices). I'm frustrated and not sure how to squeeze more performance out of these. Thanks in advance!   submitted by   /u/vitamins1000 [link]   [comments]