Following a recent interesting post that challenges the performance of higher vs lower quants and generally advised sampling parameters (namely temp), I decided that I had too much free time on my hands and ran 12 test tasks. I ran three models that work on my hardware: - q3_xxs - q3_s - q4_xs For each model I tested tasks with two temperatures : - advised sampling parameters: temp 1,top_k 20, top_p 0.95, min_p 0, presence_penalty 0.0, repetition_penalty 1.0 - temp set to 0 both in strata json and in pi models json; other sampling params removed. For each of the above model/temperature combination, I did 3 runs. Task: in-browser Tetris game with a defined set of requirements for gameplay, settings and visual effects. Quality assessment: Features delivered vs Requirements. Results in the picture. Pardon my artisanal Excel dataviz skills. Conclusions: All quants did OK-ish; not a single perfect result but all acceptable. Lower quant models don't trust themselves and generate way more tokens than higher quants. This is both from thinking and extra tool calls, tests, "actually"-s. Ktok total decode per task drops consistently with every quant step up. Higher TPS decode on lower quants does not compensate for bigger total decode and task time (including repeated tests, playwright, etc). Temp 0 task runs seem to have lower variance in task size/time; the sample is low but I would say that it's consistent with what it's supposed to do. Temp 0 may offer marginal task time decrease, better consistency and (supposedly) no impact on quality but the sample is way too low to draw conclusions. I expect that temp 0 will start underperforming in more complex tasks. Anectodally, no quality gain with higher quant; no drop either, even though it's done less testing then lower quants. So, the conclusion on performance seems to be : Run the highest quant that you can afford with your hardware, disregard the decode speed, focus on task time. I will stick with Q3s myself for now, as it's the highest quant that runs without ssd caching on my hardware, on Windows. Maybe will not be the case if I run it as a headless Linux inference box. I'm still drawn on temperature and did not validate or invalidate the findings in the post linked at the beginning. I think I'll do some more deterministic/technical tests focusing on temperature alone.   submitted by   /u/Dreeew84 [link]   [comments]