I've made a few posts recently, just talking about the hardware I've got and the model I use. Pair of P100s, Unsloth Qwen3.6 35b A3B UD Q4 K XL. There's been a ton of talk of model and hardware specific inference engines showing crazy performance numbers on models and hardware that have no business doing what they do. Inspired, I've decided to pull out Mr. Opus and see where I could push things, targeting the P100 and this model specifically, and found some pretty great gains. The work I'm about to outline is pretty shamelessly vibe coded. Blah blah, directing the model with my vision, blah blah. Vibe coded. Opus did the nitty gritty. I am wildly unqualified to explain specifics for what was done, only results and a general overview of the work. Tell opus to lay out what these cards are good at and what llama.cpp is doing relative to this model that wildly misuses the hardware, optimize optimize optimize. That's how it started at least. Then there was that post yesterday about a 3090 Qwen3.8 27b engine that was doing persistent megakernels and a reduced draft vocab, so we ended up incorporating those ideas. Truthfully, I was targeting my step father's performance in decode. He has a 306012GB+3090 setup, running Ornith 1.5. Not quite the same model, not apples:apples, but at Q4_K_M it's close enough that I'm willing to call results comparable enough to get a good understanding of where things stand. He gets ~125 t/s, unknown settings. I believe he's running through ollama. Isn't sure if he has spec decode, doesn't know his prefill. That decode number is all I have to go off of, and it was my target for this hardware. There's the semi-well-known shinbunbun p100 patches (since have been updated quite a bit so my base may not be reflective of current day performance with those patches alone) for llama.cpp that improves performance a good bit, claims +~70% on decode. This is what I started with as a baseline. My system specs as as such: Ryzen 7 5700x, stock clocks/boost 48GB DDR4@3200mhz 2x Tesla P100, limited to 175 watts each, one card stuck at PCIe gen3 x4, and the other at x16 Notably, sub-optimal. On a proper PCIe setup, performance may be slightly higher. After a few rounds of improvements, this is where we got on my system as of this early afternoon: Claude insists on calling it \"mine\", and while I did bring up the ideas and general direction, it did the actual work. And Perplexity and KLD numbers: https://preview.redd.it/b7tvi6w8y2vh1.png?width=2086&format=png&auto=webp&s=0eaf8beaed3e21ed2f9209a459f97fec4ae78e0b I think there is some work to be done with regards to some model quality. I don't think I'd run this as your main setup just yet, at least not without some testing and tuning. The numbers on paper look fine, but the model does feel a little bit different in practice. Hard to really pin it down. Exploratory? More squirrel brained. Tightening up the sampler seems to help a lot (previously temp 1.0, min p 0.05, top k 40. Is now temp 0.7 min p 0.25, probably too tight but it's working so that's where I'll keep it. Same top k), so the model can still be made to work well. I do have to note, these patches are hyper focused on the P100 running this specific model. I'm sure it'll help some other models, but I can't even guarantee it'll run other models properly at all. It is entirely vibe coded, so I cannot speak to stability. It's held up in my limited testing, but I make no guarantees. My main goal with this is to get these out there for the couple dozen other people running these cards and this model, and to maybe provide a base of patches to build on for others. Also, these patches run this still very capable model fast as fuck on decade old datacenter cards that are still kinda cheap, so I think this should be available. I'm going to be going through and making Opus do a ton of porting work if an MoE of a similar class/size comes out for Qwen4. The vast majority of the work is indeed P100 specific, and that work can be done for any model's llama.cpp implementation I'm sure. If you want to download and read the massive readme create by Opus, who understands what it made more than I do, behold the link: https://github.com/Mrinohk/qwen36-p100-patches   submitted by   /u/Mrinohk [link]   [comments]