I Spent the weekend chasing a hard power-off that hit ~7 minutes into every start of a 2-GPU Radiance vLLM tensor-parallel stack, and the answer surprised me enough that I want to know if it's a known thing. Short, short version for anyone that is impatient ... I built a new machine, a little brother for my four gpu R9700 stack that just has two R9700s in an Asus X670 Creator motherboard with an AMD 7950x, 64 gigs of ram and a 1200w BeQuiet Dark Power 14 power supply. It's running Radiance with Qwen 3.8 27b (the model isn't really relevant to the issue) In 11 separate runs as soon as the warm up happens with no power cap on the cards (each card set to 300w) the machine hard powered off. If I capped the cards to 211 watts, the machine survives and stays up. If I leave the power uncapped but disable radiance custom all-reduce and make it fail to using RCCL it also stays up indefinitely. So It definitely isn't about the total watts hit either .... it's somehow something just about this workload ... Also I did try (on both dark power units) to use the "ock" mode on the PSU also. I had a spare 1000w lian li SFX1000 power supply laying around so I swapped it in place of the dark power PSU temporarily and it did NOT fail ... so I first thought it was just a bad psu ... it happens ... so I got an advance replacement of the dark power 14 sent to me, received it today, swapped it and all the cables just to be safe ..... ran my tests again and several times in a row starting up it failed 100% of the time. I also picked up a 1200w non-sfx lian li power supply that would actually fit the hole in the case just in case the dark power replacement didn't work ... so I swapped it in and tested it and it worked several times in a row again so I buttoned up the cables and called it a day with a fully working system .... But I'm still stuck with this nagging question ... what is it specifically about these super highly rated heavy duty $300 power supplies that they can't handle this very specific load pattern consistently across multiple copies of the exact same model of hardware? Gory Detail Setup: ASUS ProArt X670E-Creator, 7950X, 2× Radeon AI PRO R9700 (300 W cards), Ubuntu. Inference stack is Radiance's vLLM build (stilldeadcode/vllm-radiance), TP=2, running Radiance's custom one-shot peer-memory all-reduce (RADIANCE_USE_R4D_AR=1) rather than stock RCCL. Each GPU reads directly from the other's VRAM over PCIe and spins until both finish. PSU: be quiet! Dark Power 14 1200 W, ATX 3.1 Titanium, both cards on native 12V-2x6. The failure: every single time, at vLLM's "initial profiling / warmup run" (the first full-size pass through that all-reduce), both cards go from ~17 W to ~270 W inside one second, then 1–2 s later the whole box loses power. No kernel message, no journal, no kdump, no pstore. The AMD FCH "previous reset reason" register is blank on the next boot, which is what you get when you pull the mains cord. The board powers itself back on 20–30 s later. The control test that points at the all-reduce: same Radiance stack, same PSU, same cards uncapped at 300 W, but with the custom all-reduce turned off and default RCCL doing the collective instead: no reset. Ran the full warmup and then 1/2/4-stream load at 101/178/305 tok/s without a hiccup. Flip the custom all-reduce back on and it dies at warmup again. Nothing else changed. What ruled out the obvious stuff: 11 for 11 failures across a dedicated wall circuit, PCIe forced to Gen4, four different R9700s swapped in and out, single-rail vs multi-rail (OCK switch), BIOS defaults, the board's supplementary PCIe header disconnected, and a brand-new factory replacement PSU with brand-new cables. The replacement unit failed at the exact same second. What makes it weird: it's not capacity. On the same PSU, each card alone pulls 324 W fine. Both at 300 W sustained is fine. Both stepping idle→300 W in lockstep with the CPU pegged at 221 W is fine. Continuous GPU-to-GPU peer copies are fine. The 1-second averages at the moment of failure are lower than loads it passed, and the RCCL run above was pulling the same per-card wattage. The other thing that makes it go away is capping both cards at 210 W with the custom all-reduce still on. So both fixes change the shape of the load, not the amount. The custom one-shot all-reduce has both dies going full compute → stall → full compute together once per layer, several kHz, in phase. RCCL's ring/tree path doesn't produce that waveform. Swapped in a Lian Li 1000 W SFX (200 W lower rating): passed first try with the custom all-reduce on, then 4 minutes of 4-stream inference at ~305 W per card. A Lian Li 1200 W is now in permanently and also passed. Same Radiance stack runs uncapped on a 4-card box with a Thermaltake GF3 1650 W, and someone else's dual-R9700 build on a Corsair HX1200i. My working theory is that the 12 V rail can't hold regulation under a sustained multi-kHz in-phase load step from two cards and either the PSU or the board's rail monitor pulls the plug cleanly. ATX 3.x transient specs are written around single big excursions, not a continuous square wave. Questions: - Has anyone seen a reset that only shows up with a specific all-reduce / collective (Radiance's one-shot, or anything similar like a custom NCCL/RCCL path or peer-memory reduce) and not with the stock collective under the same wattage? - Anyone know how to actually characterize this short of an oscilloscope on the 12 V rail? I'd love a repeatable "PSU torture test for TP inference" that isn't "run vLLM for 7 minutes and see if the lights go out." - Any other PSUs known to pass or fail this? Known good so far: Lian Li 1000 SFX, Lian Li 1200, Thermaltake GF3 1650(what is in my 4x r9700 machine running radiance), Corsair HX1200i (used in Donatos dual r9700 build). Known bad: Dark Power 14 1200 (×2).   submitted by   /u/brainchillzZ [link]   [comments]