Update: I found a mistake in my three-card numbers. Please disregard these parts of the post below: The 3x P100 row in the tokens/sec table (I had ~21 t/s generation and ~67 t/s prompt processing) The "three cards was about 4x slower than two" bullet, and my guess about why The three-card power (~160 W) and temperature (~38-51°C) figures What happened: the telemetry showed the GPU core clocks on that first three-card run were stuck at 405 MHz (idle speed) the whole time, instead of the ~1,330 MHz every other run used. I don't know the cause. It had cleared by the next day and hasn't come back. I reran the identical config and got this: Setup Prompt processing Generation 3x P100, Qwen3.6-35B-A3B (MoE) Q4_K_XL ~326 t/s ~85 t/s 2x P100, Qwen3.6-35B-A3B (MoE) Q4_K_XL ~300 t/s ~85 t/s 1x P100, Qwen3.6-35B-A3B (MoE) Q2_K_XL ~250 t/s ~76 t/s Corrected takeaways: Card count barely matters for single-stream speed. Three cards and two cards are the same, and one card gets ~90% of that (on a smaller quant). Extra cards buy you VRAM, not speed. Three-card power is ~327 W (GPUs + CPU), versus ~270 W with two cards and ~200 W with one. Temps on three cards averaged 53/56/69°C per card, with the third card (at x8) hottest and peaking at 75°C. The PCIe lane-width result is unchanged: x16/x16, x16/x8 and x8/x8 all measured the same. Lesson for me: check GPU clocks before trusting a result. Low power draw under load was the tell, and I misread it as normal. I've corrected the write-up in the repo as well. ---- Lots of you asked about the same four things, so here are the answers with real numbers. Everything below was measured while running a real agent workload (a 9-task agent battery of file edits, searches and script runs against llama-server), not just synthetic benchmarks. Cooling The P100 has no fan of its own. It expects rack airflow, and a tower case doesn't provide that. I 3D-printed a fan shroud in PETG that mounts a single 120mm Arctic P12 Pro fan to all 3 p100's. A small script reads nvidia-smi temps every 5 seconds and ramps the fan from low speed at 40°C to full speed at 75°C. The motherboard has no idea what the GPUs are doing, so its own fan curve is useless here. Under load I see about 62-64°C average and ~68°C peak with two cards, and ~71°C average, 76°C peak with one card. Cards that are mostly waiting on each other stay much cooler (~38-51°C with three). Airflow matters: after I rearranged the cards once, one of them peaked at 80°C, with the same workload and fans. I didn't feel like printing a new shroud for every setup, so I just masked off one hole when i was doing the 2 cards. It worked fine for all but one config... but it only capped out at 80°. Power Measured GPU board power plus CPU package power while running the workload (not wall power). Model is Qwen3.6-35B-A3B unless noted: Setup Draw 3 cards ~160 W 2 cards ~270 W 1 card ~200 W 2 cards, Qwen3.8-27B (dense) ~314 W Wall draw is probably 50-100 W higher than these numbers. I haven't put a meter on it yet. Each card idles around 25-30 W with a model loaded. The pstate-forcing tricks (tools like gppm, built for the P40) do nothing on the P100, so it doesn't go lower. Cabling warning: the P100 uses an EPS 8-pin, not a PCIe 8-pin, even though the socket looks like one. One PCIe 8-pin lead is only rated for 150 W and the card can pull 250 W, so I feed each card with two PCIe 8-pin leads combined into one EPS end. Don't plug a single PCIe lead straight in. PCIe lanes No NVLink on the PCIe P100, so cards talk over PCIe. My board (X79) gives x16/x8/x8 with three cards and x16/x16 with two. I tested x16/x16, x16/x8 and x8/x8 with two cards. Throughput was the same within noise. For single-stream inference, lane width didn't matter. Gotcha if you go x16/x16: on my board it disables the USB 3 ports. Mine is headless with a USB Wi-Fi dongle, and it looked exactly like a failed POST (no video, no network) until I moved the dongle to a USB 2 port. Tokens per second Prompt processing is the model reading your input; generation is it writing the answer. These are averages from llama-server's own per-request timings, one request at a time, with MTP speculative decoding on the Qwen3.6 runs: Setup Prompt processing Generation 2x P100, Qwen3.6-35B-A3B (MoE) Q4_K_XL ~300 t/s ~85 t/s 1x P100, Qwen3.6-35B-A3B (MoE) Q2_K_XL ~250 t/s ~76 t/s 3x P100, Qwen3.6-35B-A3B (MoE) Q4_K_XL ~67 t/s ~21 t/s 2x P100, Qwen3.8-27B (dense) Q5_K_XL ~123 t/s ~12 t/s Surprise: three cards was about 4x slower than two. It wasn't the quant and it wasn't PCIe width (I tested both). My best guess is the extra card adds another hop in llama.cpp's layer split, and with no NVLink each hop costs latency at batch 1. The second card mostly buys you VRAM for a better quant and more context, not speed. One card got ~90% of the two-card speed (on a smaller quant, so not a perfectly clean comparison). Software matters a lot: a set of community P100 patches for llama.cpp gave about +50% decode on Qwen3.6-35B-A3B, and MTP added another ~23% on top (measured on the 3-card setup). One request at a time only. More parallel requests raise total throughput (about 2x at 8 parallel on three cards), but I haven't retested that on fewer cards. Full write-up with the raw numbers and methods: https://github.com/4claps/houston-p100-cluster I still have a lot more testing to do, but I figured I would give yall and update.   submitted by   /u/FizzyDuncDizzel [link]   [comments]