Looking for advice on upgrading my AI server on a budget. My goal is to run Gemma 4 31B Q5_K_M (~23GB) at 16K+ context (q8), achieving at least 15 t/s. I am aware of the 26B MoE, but I want the dense model after extensive testing. Current setup: --> CPU: Intel i5-14400F --> Motherboard: Gigabyte B760M D3HP DDR4 -------> 1× PCIe 4.0 x16 -------> 2× PCIe 3.0 x1 -------> 2× M.2 PCIe 4.0 x4 (both occupied) --> GPU: RTX 3060 12GB --> RAM: 32GB DDR4 --> OS: Debian --> Inference: Ollama/llama.cpp Currently running the 31B Q5 model at 53% CPU / 47% GPU offloading, which is somewhat useable but slow (in 243, out 2.3 t/s (mtp 2.7)). IQ3_M gives a noticeable quality drop and doesn’t add any meaningful speed. My planned upgrade: Add an RTX 5060 Ti 16GB (~$750 new here in Thailand, no used available), move the 3060 to a powered PCIe 3.0 x1 riser, and use layer splitting. That gives me 28GB combined VRAM, which should theoretically be enough for the Q5 at 16K context. I'm unsure how much the x1 connection would hurt performance in a meaningful way. Other options: --> 2× Tesla P100 16GB (or similar) on a cheap X79/X99 Xeon 32-64gb ddr3 system (~$600–900 total). --> Cheaper AMD 16GB card, although mixing AMD/NVIDIA seems problematic. --> Move an NVMe drive to x1 and use an M.2-to-PCIe x4 adapter for the second GPU. Budget: $600–900, ideally less. I'm comfortable with Linux and hardware tinkering, but would prefer a straightforward setup over extreme/fragile specialized optimizations. Appreciate any real-world experience or suggestions!   submitted by   /u/Introvertosaurus [link]   [comments]