Hey guys! I have been excited to share this here. This is a project consisting of kernel optimizations for the Tesla p100 series graphics card ($80). I want to start by saying I am 17 years old and do not have a formal degree. I used Ai for a lot of this and while I understand some, I do not understand everything. Notes: My gpus are capped at 175w/250w each so these numbers may be able to be pushed higher. I also experience minor thermal throttling and sit at a nice toasty 79 degrees, which definitely effect numbers (the table above is while hot, so if you have good cooling expect 5-10% more on prefill and decode). I am using gen3 pcie with two x16 slots. Also, for anyone curious, decode numbers depicted in image were averaged from a list of questions ranging from creative writing and coding. Improved: tps went from 7-15tps at 0 context to 50-60, 260k context went from 2-4tps to 30-35, prefill went from 220 tps at 0 context to 350, 260k context went from 40tps (as far as i remember, i never really measured cause it was too hard) to 110tps, fixed fp16 math errors by using some mixed fp16/fp32 math operations so rounding errors were eliminated, and merged as of sept 22 so it should support qwen 3.8 flash architecture This setup is somewhat flag specific (ie: (-c 262144 -b 32768 -ub 1024 -np 1 \) without -b 32768 mtp becomes overloaded and drops acceptance to near 0 at full depth) so keep that in mind while setting up. One more thing, I took regression very seriously in this. Math had to be more accurate or byte identical or it would fail tests. Build, flags, math proofs, and anything else you may need will be linked below. Enjoy guys! I would love your feedback on this and am looking at pull requests. Github: https://github.com/Kmic-68/llama.cpp/tree/p100-optimizations Details for build, math proofs, etc: https://github.com/Kmic-68/llama.cpp/tree/p100-optimizations/p100-docs   submitted by   /u/Kmic68 [link]   [comments]