Now that I've gotten a lot of advice on risers, I'm trying to figure out the best long-term setup for my local inference workloads without spending a fortune on hardware I might not actually need. I currently have: Component Main inference server Helper server Spare hardware GPU 2x RTX 3090 24GB (250W PL) 2x RTX 3060 12GB 2x RTX 3090 24GB + 1x RTX 3060 12GB CPU Ryzen 5900X i3-10320 EPYC 7302P Motherboard ASUS Crosshair VIII Hero X570 ASUS B460-F - RAM 64GB DDR4-3200 (4x16GB) 16GB DDR4 (2x8GB) - PSU Corsair HX1500i Corsair RM850 Corsair RM650x The two options I'm considering 1. Buy an SP3 motherboard (probably H12SSL-I/NT or equivalent), 128–256GB ECC DDR4, and risers to consolidate all 7 GPUs (132GB VRAM) into one system. More PCIe lanes and memory bandwidth/capacity. Could also expand RAM further later. This is obviously not cheap, probably looking at $2500-3000 for a motherboard, cooler and memory. 2. Run 4x 3090 on the X570 at PCIe 4.0 x8/x8/x4/x4 using risers/M.2, and figure out how to fit all 3x 3060 into the B460 system. Much cheaper, but with obvious PCIe and RAM limitations. Workloads are primarily agentic coding using STRATA, vLLM etc, but also a lot of integration into my own apps/infrastructure. I'm interested in running multiple models concurrently, potentially something like: STRATA / large MoE as the planner/orchestrator/reviewer Qwen 27B dense as the implementer Smaller models for task routing, embeddings, reranking, voice-to-text and other background services (these currently run on the 2x 3060s) Is consolidating everything onto EPYC actually worthwhile for these workloads, or would I get better practical performance/value from two separate systems? How much would the 7302P hold back inference, especially with multiple concurrent models or CPU-offloaded MoEs like those supported by STRATA? Would 128–256GB RAM meaningfully change what I can run compared to keeping everything GPU-resident? Would you dedicate the 3090s to larger models and the 3060s to smaller/background models, or distribute things differently? Anyone running something similar who has compared a consolidated EPYC setup against multiple consumer platforms? Anything else you'd recommend instead? Used server motherboards and ECC RAM aren't necessarily cheap here, e.g around $1000-1200 just for a used H12SSL-NT. The EPYC CPU itself is already paid for, but the motherboard, RAM and risers would still be a fairly significant investment. Not interested in spending thousands just to consolidate everything if the actual inference benefits are marginal. Particularly interested in firsthand experience and real benchmarks rather than theoretical PCIe bandwidth numbers.   submitted by   /u/infectiousstupidity [link]   [comments]