Since my last post, I've been thinking about different options for dynamic performance degradation, trying to squeeze as much high-quality inference out of my GPU as I can. Over the weekend I read this really interesting paper: Cache-to-Cache: Direct Semantic Communication Between Large Language Models. In it, the authors describe running multi-llm agent systems. But rather than having agents talk to each other through a harness+tool calls+messages, they had agents pass context to each other by fusing one agent's kvcache directly into another's. Assuming this is possible, you could imagine this being a faster, more complete way to pass context between agents: rather than one agent producing a summary/handoff message, you literally just rip out its working memory and graft it onto the target model. They go on to describe how they do this, the TLDR being they trained a small neural network to be able to "convert" between the source and target model's internal representations, allowing them to fuse kvcaches of models of differing size and even architecture. --- This got me thinking: what if I wanted to reuse a kvcache between different quantizations of the same model? I mean, same architecture, same training process ... shouldn't they be compatible, even without training a 'converter'? And what would happen if I started inference with a high-precision quant, then swapped in a lower-precision quant to 'take over' when running low on device space? Could I get better results than just running the lower-precision quant from the start? Spoiler, the answer to all of this is yes (on the benchmarks I ran)! I detail the specific experiment I ran below. --- Methodology I generated difficult NIAH-style tasks at different context lengths, and had 5 different Qwen3.8 quantization strategies battle it out! For these tasks, I used three different quantizations of Qwen3.8-27B, each made by unsloth: - UD-Q6\_K - UD-Q4\_K\_XL - UD-IQ3\_S Strategies From these quants, I defined three static-quant strategies to run tasks against: IQ3\_S: f16 kvcache, max ctx 196,096 Q4\_K\_XL: q8\_0 kvcache, max ctx 183,296 Q6\_K, f16 kvcache, max ctx 175,104 Note that the Q3 and Q4 strategies have ctx windows sized for a 24 GiB GPU, while the Q6\_K case requires > 24 GiB to run. This is to evaluate how closely static quant strategies on a small device measure up to a static quant strategy on a larger device. The idea is to see if dynamic quantization strategies can make up some of that difference! Speaking of, I defined two dynamic-quant strategies to compare against each of the small precision static-model cases. These dynamic-quant strategies also both feature ctx limits sized for a 24 GiB GPU. IQ3\_S Comparison For this strategy, I ran the tasks against a multi-quant strategy with a worst-case model quantization of IQ3\_S: - Start task with Q6\_K, f16 kvcache, max ctx 54,272 - Then swap in Q4\_K\_XL, f16 kvcache, max ctx 109,312 - Then swap in IQ3\_S, f16 kvcache, max ctx 192,096 In the data, you can see this strategy labelled as Q6→Q4→Q3, f16 KV. Q4\_K\_XL, q8\_0 kv Comparison For this strategy, I ran the tasks against a multi-quant strategy with a worst-case quantization of Q4\_K\_XL, q8\_0 kv: - Start task with Q6\_K, f16 kvcache, max ctx 54,272 - Then quantize the model's kvcache to q8\_0. Max ctx: 91,136 - Then swap in Q4\_K\_XL, f16 kvcache, max ctx 109,312 - Then quantize the model's kvcache to q8\_0. Max ctx: 183,296 (For the mid-run quantizations, I used the hot-reload method described in my last post. The f16q8\_0 conversions are handled by the same llama.cpp fork.) In the data, you can see this strategy labelled as Q6/f16→Q6/q8→Q4/f16→Q4/q8. Tasks I generated dozens of unique NIAH ("needle in a haystack") tasks, which direct models to parse large volumes of input text and follow specific instructions scattered throughout the text to retrieve a secret value. (h/t gkamradt/needle-in-a-haystack for some of the source material) I went with NIAH because it felt like a reasonable way to evaluate coherence for the multi-quant strategy. Each task requires the model to reason through a sequence of 'steps' buried inside distraction text, so a model with a transplanted kvcache would need to be capable of picking up the train of thought precisely where the source model left off. Also, NIAH doesn't require a complicated test setup, and the answers are objectively right or wrong. For each task and test case, I measured the following: * Result (correct/incorrect) * Total tokens generated * Total time taken I included time taken despite each quant having a very similar prefill/decode speed because I wanted to demonstrate that the multi-quant approach does not take noticably longer than running a single-quant strategy. Transferring the kvcache from one quant to another means we don't need to repeat prefill! Results In total, the benchmark tasks I laid out represented 180 distinct runs, and which took my GPU 14h, 26m to complete. The biggest offender here was the IQ3\_S/f16 strategy. Especially for the heavier tasks, it consistently generated upwards of 50k reasoning tokens, and all-too-often completely max out its context window (\~196k) before failing to ever generate a response. Still, it holds up reasonably on the shorter tasks, even managing to score higher than Q4\_K\_XL, q8\_0 in terms of agreement with Q6/f16. Shoutout xhigh reasoning, I guess! Speaking of agreement with Q6/f16, to me that was an important metric to track, because I wanted to compare how much closer a dynamic approach got to approximating the high precision reference. Agreement with Q6/f16 _SEE IMAGE 1_ https://preview.redd.it/739ykn0h8qrh1.png?width=1057&format=png&auto=webp&s=959e05ef0110a00d687bd288cb67c254725fb27c This graph shows the number of tasks whose final answer is exactly identical to the Q6/f16 result, even if that answer is incorrect. The motivation here was to identify whether a dynamic quantization strategy could approximate Q6/f16, and I would argue that this graph is a strong indicator that it can! In both cases, the dynamic quants match the Q6/f16 model's results much more closely than their static counterparts. Overall, the path that avoids IQ3_S ends up far closer to Q6 at high context, which isn't too surprising! I don't want to put too much weight on task correctness, hence the focus here on "agreement with Q6/f16." This is because I'm not convinced my NIAH tasks are representative of performance at large. (That said, I do include task correctness results below, in case you're curious). Avg Inference Time and Avg Output Tokens _SEE IMAGES 2 and 3_ https://preview.redd.it/i7tmdbdm8qrh1.png?width=1057&format=png&auto=webp&s=aa475a88ee31d5f334ccb541be8f3594598572b3 https://preview.redd.it/i4klmw0l8qrh1.png?width=1057&format=png&auto=webp&s=e72d0d3f54e12012d999d5456135a07d6bedd2ba These graphs show the arithmetic mean of inference time (seconds) and total output tokens across completed task seeds, including incorrect and context-exhausted runs. I particularly wanted to highlight inference time, because for the dynamic strategies, it includes the time to swap out model weights and quantize the kvcache! I think this is a nice demonstration of the benefits here -- inference time across tasks really doesn't get worse, just because we're doing fancy dynamic quantization strategies. This is because: - When swapping model weights (e.g. Q6->Q4), we're doing a direct KV cache transplant, straight up moving the kvcache from one quant to another. - When quantizing an existing model's kvcache (e.g. Q6/f16->Q6/q8), I'm using my fork of llama.cpp that hot-reloads a live model's context/runtime and automatically converts between kvcache precisions. In short, in both cases, there is no need to repeat prefill! After transitioning, the models continue prefill/decode precisely where they left off. Task Correctness Here's a table of task correctness across all strategies/runs. I've split the results by "lowest model precision used" to make the static-dynamic comparison easier. Worst Case IQ3\_S: Strategy 10k 25k 50k 75k Q3/f16 6/10 9/10 2/10 4/10 Q6->Q4>Q3 7/10 7/10 7/10 4/10 Result: dynamic quant beats static in 2 cases, ties once, and loses once. Worst Case Q4\_K\_XL, q8 kv: Strategy 10k 25k 50k 75k Q4/q8 7/10 8/10 5/10 4/10 Q6-:>Q6/q8->Q4->Q4/q8 7/10 7/10 7/10 7/10 Result: dynamic quant beats static beyond 50k context, and mostly breaks even before. Overall: Here I compare the dynamic strategies directly against the reference, removing the 10k and 25k tasks, because below those levels the dynamic strategy is literally just running Q6/f16. They're identical every time. Strategy 50k 75k Q6->Q4->Q3 7/10 4/10 Q6->Q6/q8->Q4->Q4/q8 7/10 7/10 Q6/f16 6/10 8/10 Results: I don't think there's much to draw from these results, except that the IQ3_S quant really falls apart at high context. This table demonstrates why I didn't take task seriousness too correctly. Taken literally, it suggests that Q6->Q4 and Q6->Q6/q8 are superior to Q6/f16 at 50-75k context! Conclusion I'm quite happy with these results, overall! Although this benchmark isn't perfect, for my purposes I am more than satisfied that dynamic model quantization is a good way to offset the typical precision loss that comes with hardware constraints. I geared my tests mostly around pushing the limits of a 24 GiB GPU, because it's easier to compare against a reference which can only be run on a 32 GiB GPU. As a next step, I'm going to integrate this into my inference setup and see how well this holds up when activating all the bells and whistles (namely, speculative decoding and mmproj, neither of which were enabled during these benchmarks). My intuition says the tradeoff to get right when using these strategies for IRL inference is to avoid stepping model quantization down too frequently. While I think coherence would be fine, at some point the time required to swap out weights will become noticeable. So, I think I'll try and set things up so that I create large "tranches" of context where the model runs unchanged for ~40-50k tokens. IMO the Q6/f16->Q6/q8->Q4/f16->Q4/q8 strategy is already a great example of this. kvcache reloads take much less time than model reloads, at least with my current llama.cpp changes. Maybe I could work on that in the future!   submitted by   /u/wadeAlexC [link]   [comments]