background so, I'm director of scientific computing at a cancer center and have just received some interesting news: a new lab has between a $300-400K budget to pay for GPU hardware to run their AlphaFold 2 workloads. some of the pertinent characteristics: they need a fair amount of VRAM per job, but not that much; they start to hit diminishing returns after around 50GB. it is very compute intensive but not very data intensive; input data is small and output data is even smaller (literally just some text files). caveat: they end up having millions of tiny files in a fairly varied directory tree, so IOPS for storage will end up being a thing to think about (this is probably fine; we can script around this with prolog and epilog scripts moving data in and out of node-local storage and in/out of our HPC Isilon). what we currently have a 70-node cluster of Dell R7525s, 3rd-gen EPYC, 1TB DDR4 RDRAM per node, one (1) A30 per node (can you tell this machine was built shortly before the AI boom?). we do have a project on the docket to have the A30s pulled out of 35 of the nodes and installed next to the other half on the same NUMA node, such that we have 35 nodes with two A30s and 35 empty nodes. each node has 3 available PCIe slots, two on CPU 1 and one on CPU 0. I've run a bunch of tests with Qwen3.8-27B running in TP=2 across two of the cards on CPU 1 and got very nice numbers back, so I know that's a usable strategy for inference. haven't tried Strata yet although it's on the agenda. fairly vanilla system otherwise, Slurm 24.something, Bright 10.something, somewhat shitty networking (2x25Gb LACP to each node). networking is a planned upgrade next year to at least 400GbE. other context here: we're about to start an assessment, working with a consulting firm on, more or less, building the future of AI at the company. we (my group) have very strong opinions on how to do this and from what I can tell, the consulting firm agrees with us. that will mean a big budget for a large system refresh... at some point in the medium-term future, we do not know when. frankly, my inclination is to hold onto the R7525s until they start dying, because they serve our CPU-oriented workloads perfectly well and aren't having any issues to speak of -- I would rather spend refresh money on the networking, possibly some dedicated data-management-inclusive HPC storage, and of course greatly upgraded GPU capability -- like, training gear, not just RTX Pro-tier things. what I'm trying to figure out now what the most efficient way to spend this money is. why I'm asking here while I already have a pretty solid idea of how I want to do this, I know plenty of you are Sickos™ when it comes to how well hardware plays with LLM workloads and probably no small number of you are familiar with using GPUs for other scientific workloads as well. the goal with this build-out is to maximize the utility of the hardware (and, by extension, the utility of this money) not just for this lab's work but for everyone else's as well, and everyone else's work very much includes running lots of local models as fast as possible. the options purchase a single system, self-contained, with 8x RTX Pro 6000 Blackwell (two of these would blow the budget) or 8x RTX Pro 5000 72GB Blackwell (might be able to squeeze two). politically this might be easier. it will be strictly fewer cards than option 2 because we'd have to spend money on CPUs and (gulp) system RAM. suboptions! we buy as many RTX Pro 5000 Blackwell 72GB cards -- just the cards -- as we can afford, and then, put one per system in as many nodes as we can afford to buy cards for, double them up like the A30s for 144(!)GB VRAM per node. unless there are mitigating circumstances/weird factors/gremlins I'm not thinking of, I lean heavily toward option 2.2. it preserves the available horsepower for the new lab's AlphaFold 2 work while also giving us a very, very nice platform for running great big MoEs in TP=2 on the pair of 5000s. potential "goddammit" factor we are REEEEEAL close to our overall power consumption limit in our datacenter. each one of the cards we're considering here only has a 300W TDP, but you're still talking about adding, in the best case, maybe 20 cards (24? something like that), meaning 6kW additional load. I don't *think* this will put us over the edge but we're already definitely pushing it. we *could* power-limit them if we need to scoot under a line, so this might not be that big a deal. what am I forgetting? open question to alla'y'all who read this far (congratulations, have a cookie 🍪).   submitted by   /u/starkruzr [link]   [comments]