I calculated KLD and perplexity for Qwen3.8 27B models that fit 16GB VRAM: See comments for the benchmarks. Reddit filters keeps deleting the post. Conclusion: q5_1/q5_1 is almost always better than q8_0/q4_0 or q4_0/q8_0. There is very little difference between q8_0/q4_0 and q4_0/q8_0 q4_0/q8_0 is sometimes better than q8_0/q4_0. So it's safer to just go q5_1/q5_1. Going up a model quant and running at q4_0/q4_0 is almost always better than running the quant below at f16/f16 in terms of KLD. So basically pick the highest quant you can at q4_0/q4_0. Then if you have room to spare increase KV quant. Note this applies to lower quants. If you are going from q6->q8 model quant it might be bigger gains increasing KV quant, due to diminishing gains going up model quants. Also this does not apply to other models like Gemma where you have to run f16 KV. Bonsai and swift quants are included but it does not really make sense to calculate KLD since they are finetunes and the KLD will differ significantly.   submitted by   /u/Fancy-Snow7 [link]   [comments]