I saw this quite a fair bit with 3.8 27B Q4_K_M, which is reduced greatly by Swift. I tried a very low quant of Swift's QFN (IQ2-XXS) and the second guessing was much worse. I've read quite a bit that 27B is very thorough and that's how it achieves its high scores relative to its parameter count. Is this particular behaviour part of its model, or an artifact of its quantization? Would we not see this behaviour on BF16 GGUFs? Assume a high enough context that compaction is not applicable.   submitted by   /u/mailto_devnull [link]   [comments]