Some results for those using GGUFs and llama.cpp: GGUF quality: MACaQ vs. Unsloth and Bartowski Qwen3-4B (IQ4_XS) Quantization GGUF size Mean KL ↓ MACaQ 2.260 GB 0.02804 Bartowski 2.271 GB 0.03443 Unsloth 2.271 GB 0.03663 23.5% lower KL divergence than Unsloth and 18.6% lower than Bartowski, at virtually identical file sizes. Qwen3.6-35B-A3B (IQ4_XS) Quantization GGUF size Mean KL ↓ MACaQ 18.86 GB 0.018696 Unsloth (UD-IQ4_XS) 17.73 GB 0.020954 Bartowski 19.70 GB 0.021927 Qwen3.8-27B (IQ4_XS) Quantization GGUF size Mean KL ↓ MACaQ 14.96 GB 0.036784 Bartowski 15.48 GB 0.104170 ~65% lower KL divergence than Bartowski! Supported GGUF formats (llama.cpp) Weight bits Formats 2-bit IQ2_XXS, IQ2_XS 3-bit IQ3_XXS, IQ3_S 4-bit IQ4_NL, IQ4_XS Code, llama.cpp fork, and benchmarks: https://github.com/huawei-csl/MACaQ Cool to see Huawei working on such stuff and sharing some of their internal tools that are helpful for us. Maybe they can build an affordable GPU for us local LLM folks sometime 😂 They also report quantizing a 122B-parameter model in 64 minutes using just 32.5 GB of GPU memory, which is interesting for a model of that size. I did not try that. Edit: there is a paper on it https://arxiv.org/abs/2611.10649   submitted by   /u/acluk90 [link]   [comments]