Following my previous post on how KLD is misunderstood, there were a few claims that KLD is the best metric you can use and some other comments were asking, "If we stop using KLD, what should we use instead?" I'm gonna explain both a bit more here. Assume you're a lab that does distillation. In the distillation process, you choose a teacher model and a smaller student model that you want to learn from the teacher. This learning starts with optimizing a variant of KL loss (it could be KL itself, reverse KL, or JS loss) between the outputs of the student model and the teacher model. After the optimization ends, according to Reddit, you must be done. You have a model that has the best possible KLD against the teacher model, and what could be better than that? But no, the training can still continue. After that, there can be stages of SFT and RL to improve the performance of the model. These stages improve what really matters, and that's the ability of the model to solve different tasks. And during these stages, the KLD you had can become even worse, but the student model actually becomes better. This could be similar to the process that PrismML uses for their Bonsai models. They can give you a KLD number that blows your mind but instead their model probably goes through additional stages of training that actually make the model better but the KLD worse. And these additional stages could be the reason that they treat Bonsai as more than just a quantized version of the original model. Benchmaxxing can happen during these stages when, instead of optimizing on some held-out tasks, you optimize your model on tasks that come from popular benchmarks. So my whole point is that lower KLD doesn't mean better ability to solve different tasks. And there is no single loss, at least not one we're aware of, that solves this issue completely because otherwise we could just use that loss in distillation instead of putting our money into several other stages of training. So what's the solution here? The first thing I want to make clear is that I'm not saying you shouldn't use KLD at all. It can work as an initial filter to see which quantized models are worth putting your time into, especially if one model is way off in KLD. But it has to be calculated on representative data, and data generated by the base model itself can be particularly useful as I explained in my previous post. Otherwise, the number might tell you very little about the actual performance of the model. The second thing is that you don't need to rank every quantized model in the universe and I don't know where that desire comes from. You want to find a model that works for your use case. You can store some of your day-to-day tasks, test those tasks on a few quantized models, and choose the one that works best for your tasks. I know it's not an easy process. Grading can be challenging, and organizing and creating a private benchmark can be hard as well. I get the challenges, but you do it once, and then you can easily test and choose future quantized models. I hope this clears up some of the points that were missing from my previous post and the comments helped me clarify them further.   submitted by   /u/Positive-Resource922 [link]   [comments]