KV cache quantization has been a hotly-debated topic for this community for some time, with opinions ranging from "never quantize" to "4 bit is fine". Since the anecdotal evidence is rather inconsistent, I decided to quantify how much KV cache quantization affects the model through KLD, which is a good measure of how the model's output distribution is changed. I tested 35 weight quants across llama.cpp, exllamav3, and vLLM, each with 16-bit, 8-bit, and (where supported) 4-bit KV cache. I also reworked my KLD dataset and measurement methodology this time, to include a mix of agentic tool-use (generated by Qwen3.8 27B at full precision) and a variety of texts in different domains. The current dataset now stretches up to 64k token contexts per row. TLDR; 8-bit KV caches work well. In llama.cpp and ExLlamaV3 the cache size is halved, and there’s essentially no difference in KLD. There’s a slight difference in vLLM, but it’s still a great tradeoff for people lacking VRAM. A 4-bit cache has a measurable impact. It’s a big hit for ~Q8 weights, but it could potentially be worth it if you’re using Q3 or below. Still, not recommended. Only the 4-bit cache impact grows with context. In the tested long documents, the added KL is about 15-30% higher at 16-32k tokens than in the first 4k. The blog post has interactive charts and tables with all of the data and results if you're interested. How to read the numbers Each quant is compared token-by-token with an unquantized bf16 copy of the model. KL divergence measures how far the quant’s next-token probabilities drift from the original (0 means no change). Top-1 agreement is how often both models pick the same most likely token. For a reference point, I ran the unquantized model in both llama.cpp and vLLM and compared the two. They differ by a KL of 0.0004, which essentially amounts to two (correct) implementations doing floating-point math in a different order. Thus, you can compare the KLD numbers here for quants running in different engines without issue. Note: exllamav3 quants were measured with the bf16 llama.cpp reference logits. Weight quantization KL divergence vs. loaded weight size, 16-bit KV cache: The blog post has an interactive chart and the full KLD numbers in a table, check that out if you're interested. The vLLM checkpoints still do worse As in the Qwen3.6 benchmarks, every vLLM checkpoint is beaten by a smaller GGUF or EXL3 quant. Some of that is due activation quantization. NVIDIA’s and Unsloth’s NVFP4 checkpoints run the MLP layers as W4A4 (4-bit activations) and attention as FP8 W8A8, and Qwen’s FP8 is W8A8 throughout. I reran NVIDIA’s NVFP4 and Qwen’s FP8 through vLLM’s Marlin kernels, which use the same quantized weights but keep the activations in bf16: NVFP4: KL drops from 0.0256 to 0.0117, more than halving it. FP8: KL drops from 0.0039 to 0.0028, about 30% less. Even then it only ties UD-Q5_K_XL, which is 8.7 GiB smaller. KV cache quantization KV cache Per token At 64k context Saved vs. 16-bit f16 / bf16 64 KiB 4.00 GiB - 8-bit: llama.cpp q8_0, EXL3 8-bit 34 KiB 2.13 GiB 1.9 GiB 8-bit: vLLM fp8_e4m3 32 KiB 2.00 GiB 2.0 GiB 4-bit: llama.cpp q4_0, EXL3 4-bit 18 KiB 1.13 GiB 2.9 GiB Each dot below is one weight quant with an 8-bit or 4-bit KV cache. Its height shows the KL increase compared with the same weights on a 16-bit cache. Zero means no change; higher points show a larger increase. I computed the differences per prompt on identical tokens. KV cache penalty vs loaded weight size: Open markers show 8-bit KV; filled markers show 4-bit KV. Colors identify the engines. The dotted line marks the KL difference between the two bf16 reference engines. vLLM has no 4-bit KV cache option. 8-bit is essentially free in llama.cpp and ExLlamaV3. For every quant, q8_0 and ExLlamaV3’s 8-bit cache move KL by less than 0.00012 in either direction. Some quants even measure slightly better with it, which indicates it’s within the noise of the numerics. vLLM’s FP8 cache shows a slight KLD increase. It adds 0.0004 to 0.0006 on every checkpoint, which is still extremely small, but it shows up on nearly every prompt. It may be due to turning on FP8 cache switching vLLM’s attention backend from FlashAttention to FlashInfer. llama.cpp’s 4-bit cache adds a roughly fixed penalty. Across the GGUF quants tested, q4_0 adds +0.0016 to +0.0027 KL. This increase has a larger relative effect on more precise weights: On a Q3-class quant (KL around 0.025-0.035) it adds 6-10%. On UD-Q4_K_XL it adds about 39%. On Q8_0 it adds about 390%. KL goes from 0.0007 to 0.0034, which is worse than UD-Q5_K_XL with a 16-bit cache. EXL3’s 4-bit cache penalty seems to depend on weight precision. It adds only +0.0002 to +0.0004 KL with 2.0-2.5 bpw weights, about +0.0021 to +0.0022 with 3.0 bpw weights, and +0.0026 to +0.0028 with 4.0 bpw weights. At 6.0 and 8.0 bpw, the penalty reaches about +0.0032 to +0.0033. The increase is therefore not constant across the EXL3 quants. 4-bit cache penalty grows with context The 12 long documents in the set are scored at every position, so I can check whether the 4-bit penalty depends on how deep into the context a token is. It does in fact grow slightly. Position Q8_0 Weights + q4_0 KV UD-Q5_K_XL Weights + q4_0 KV EXL3 8.0 BPW Weights + 4-bit KV First 4k tokens +0.0035 +0.0030 +0.0041 4k–8k +0.0037 +0.0033 +0.0044 8k–16k +0.0040 +0.0036 +0.0047 16k–32k +0.0041 +0.0038 +0.0050 32k–64k (one document) +0.0044 +0.0038 +0.0053 There’s about 15-30% more damage at 16-32k context than in the first 4k tokens. The same quants with a 16-bit cache are unaffected across these context lengths, so it’s a pretty clear indication of the 4 bit cache damage. I would expect 128k+ context lengths to be even worse. Model quantization KLD vs KV cache KLD: KLD due to KV cache quantization likely has different quality tradeoffs compared to KLD from weight quantization. Anecdotal evidence suggests that KV cache quantization can be more harmful to model performance at long context lengths compared to weight quantization, so be wary of using 4-bit KV cache. Final chart with weights + 64k KV cache versus KL divergence: The line highlights the recommended quants, excluding 4-bit KV. Diamonds show 8-bit KV; faded circles and triangles show 16-bit and 4-bit KV. Memory includes weights and the KV cache at 64k context. Methodology Each quant was scored against an unquantized bf16 reference: GGUF and EXL3 quants scored against Bartowski’s bf16 GGUF in llama.cpp. vLLM checkpoints scored against the official bf16 safetensors in vLLM. KL is computed the same way as my previous benchmark. The top 200 log probabilities are taken from both models at each position and a lower bound on the full-vocabulary D_KL(P_reference ‖ P_quant) is computed. On average, the reference put 0.0036 of its probability outside its top 200 tokens. The KV deltas are paired, meaning they’re computed from per-prompt differences on identical tokens. Checkpoints ran with their declared settings, including any activation quantization (except the two weight-only, no activaton quantization reruns of Unsloth’s NVFP4 and the official FP8 quant).   submitted by   /u/Hefty_Wolverine_553 [link]   [comments]