With Mixture-of-Experts (MoE) models becoming increasingly popular, I decided to build a budget-friendly AI workstation, betting on the possibility that MoE architectures will become the new standard for local LLM inference. My initial realization was that a single AMD Radeon AI PRO R9700 with 32 GB of VRAM could potentially cover a surprisingly large portion of my inference needs. So I started looking into cheap DDR4 ECC RAM to store inactive experts, while keeping the active experts in VRAM. To figure out the ideal configuration, I plotted some of the most popular open-weight MoE models, comparing their active and inactive parameter counts against the VRAM and system RAM required to run them at Q4 and Q8 quantization. The data shows a fairly consistent linear trend between active and inactive model weights. The fitted slope is approximately 0.0565, suggesting a rough ratio of 1 GB of active VRAM weights for every 18 GB of inactive weights stored in RAM. While this is only an empirical trend, it provides a useful guideline for deciding when adding more RAM stops making sense without also upgrading VRAM. For my particular setup, 384 GB of DDR4 RAM paired with 32 GB of VRAM appears to be the sweet spot. According to this trend, 384 GB of inactive weights would correspond to approximately 21.7 GB of active weights, leaving some VRAM headroom for KV cache, activations, and other runtime overhead. Going beyond that, especially toward 512 GB of RAM, would increasingly push the average model’s active working set toward the limits of my 32 GB GPU. Of course, actual results depend on the model architecture, expert routing, caching efficiency, and inference backend. Still, this suggests that a relatively inexpensive EPYC workstation with 384 GB of DDR4 and a single 32 GB GPU could be a compelling platform for running a wide range of large MoE models, potentially including many Flash-class models at Q8.   submitted by   /u/Numerous-Ad6217 [link]   [comments]