So I've been trying to get my own thing going again after quite some time of having given up on local llm but now models are strong as ever and I need agentic ai for some stuff and usage gets expensive. So my machine is a g292-z20 with 8x p100 pcie running manjaro on the bare metal with a self compiled llama.cpp. Model is Qwen 3.8 flash next Q4 the unsloth one. Since I want to use it for agentic I really want good concurrency. With 1 I get about 24-20tps generation I tested prefill before with large files to be about 400 to a little over 500. If i use MTP I can even get a solid 33tps with concurrency of 1. However when I do without TPS with 16 slots I get good performance with a maximum of total about 80tps at 11 concurrency and then it falls of sharply. But when I change the number of slots I always get concurrency number where it drops off afterwards it's pretty good again before dropping off again. Best I got was 20 slots at 13 concurrency I got 91 total. But before and after it was horrible like 30ish total or so. And it's static per slot number but changed wildly and unpredictability with different slot numbers. What could it be? I guess I got pretty good performance for the size of the model already on that old hardware but I really feel like I'm missing out on some optimization here.   submitted by   /u/ILoveDangerousStuff2 [link]   [comments]