Hi r/LocalLLaMA, we're releasing Boole-32B-A2.5B-Preview, an early 1.8-bit version of Recursive's flint. With mixed-precision quantization, expert pruning, and custom kernels (kernel changes in attached figure), we made the model 9x smaller than BF16, and 2x faster than INT4. After releasing our 1-bit LLM (Boole-1.7B), we heard a lot of requests for both faster and larger MoE models - so this is our first step in that direction. Try it out, and let us know what you'd like to see next! Weights: https://huggingface.co/precision-numerics/Boole-32B-A2.5B-Preview-GGUF Getting started: https://github.com/PrecisionNumerics/boole-demo Our llama.cpp fork: https://github.com/PrecisionNumerics/llama.cpp https://preview.redd.it/t0l4rqpxlc1i1.png?width=768&format=png&auto=webp&s=566e4617674c8a47dcd26a2c60e06e0e40a4a72e https://preview.redd.it/m1b44w2ylc1i1.png?width=768&format=png&auto=webp&s=a9625ef9811e51f833315da69cbdcf7e1871032a https://preview.redd.it/qsvvdihylc1i1.png?width=768&format=png&auto=webp&s=cd73999284597fa20689a35f4f28a97b6779b165   submitted by   /u/d77chong [link]   [comments]