Dynamic Quantiser TLDR - Makes dynamic and/or custom quants of ggufs. Doesn't need data, just gguf file + llama.cpp. Minimises cosine deviation - highly correlated with minimising KLD. Fast, much better than standard quants, not as good as Unsloth on pure text, maybe as good on mixed inputs/code. Free, open source. Intro Allows anyone to make their own dynamic quantisations of LLM .gguf files at ANY desired size or quality. Automatic, fast and fairly optimal - considerably more accurate than standard K quants. Should work with any llama.cpp compatible .gguf file (bf16/f16/f32). Chooses tensors at mixed quantisation levels to minimise whole-model cosine deviation. Also lets you make your own custom quantisations by picking layers/groups at different quantisation levels. Motivation Many local LLM users are VRAM constrained, and this tool lets anyone make exact-size quantised models that are considerably better in accuracy than standard Qx_K_M quants. Whole-model cosine deviation is highly correlated with KLD (a 0.99 Pearson correlation) - a good measure of quant quality. Once the initial table has been built, solutions are just seconds. How it works There are 3 different algorithms in the program: Make Dynamic Quant - the most accurate, full solution. It minimises whole-model cosine deviation for a fixed file size, choosing the best quantisation level for every individual tensor. Requires a target file size. Minimise cosine deviation - minimises whole-model cosine deviation, but not at tensor level; instead it uses a much more approximate solution at layer/group level. Useful for making your own custom quants - easy to tweak from a decent base solution. Also requires a target file size. Minimise disk size - again at layer/group level (not tensor level), this does the same thing as 2, but minimises disk size instead, subject to a fixed cosine deviation. Requires a target cosine deviation. Technical For every tensor and every quant level it precomputes the dot product with the original weights (S), the squared norm of the dequantised copy (Q) and the byte count (C). Whole-model cosine deviation is then 1 - S / sqrt(AQ) and the goal is to choose a quant level per tensor that minimises this deviation subject to a total file-size budget. That ratio-of-sums objective is optimised with Dinkelbach's algorithm and a separable (multiplier-based) Lagrangian inner loop, polished with a multi-start local search over single-tier and pair-swap moves. Benchmarks I made q3, q4, q5, q6 size quants for Qwen 3.8 27b to compare to standard and Unsloth, with and without imatrix. Of course you can make quants at any size you like - these sizes were picked to compare to standard quants. WikiText (sample): https://preview.redd.it/ilozyrmo53rh1.png?width=1500&format=png&auto=webp&s=b3c7908f4297a14787a8d1110a18706d532dcc4c Unsloth crushes here. Mix dataset (Bartowski's calibration v5): https://preview.redd.it/zp7dklgs53rh1.png?width=1500&format=png&auto=webp&s=6ae971a9d7d68ccf69f995ea34a9ed440238e029 Weirdly seems to beat Unsloth dynamic 3.0 on some quants here. Limitations Made with AI. The idea is mine, the code is written by AI (I know people don't like that here but I cannot code well). I am a noob Windows user. I do not have Linux and have not tested on Linux. Local Qwen says it will work though, so rest assured it definitely will! /s Not fully tested on every model; there will be bugs. It seems to work well on Qwen, Gemma and a random sample of other models I tried. I will fix bugs if you report them. The initial table build is slowish. On a Ryzen 5 3600 it takes ~25 mins to build it for Qwen 3.8 27b from the 53GB source file. This is a one-off build though. You have to download the whole bf16 file. Minimising cosine deviation does not equal minimising model errors. Not as good as SOTA dynamic quants like Unsloth dynamic 3.0. It's more for making your own custom stuff + the code is open source. Doesn't use IQn_XS style quants. I actually have coded some of these and they are easy to add. However, because they seem more efficient for cosine deviation minimisation, they get used a lot. This slows down inference too much as they are slower to compute so I omitted them for now. They do push down KLD a fair bit more though. Undecided. Requirements are minimal — just things you probably already have (Python, C++, llama.cpp). See the GitHub README for setup and downloads. For Windows it's just a zip file with a .exe in it. GitHub: Dynamic Quantiser   submitted by   /u/animatedata [link]   [comments]