I get a lot of hate whenever I tell people I run a Q4 quant of Qwen3.8 27B with 262k KV 80t/s with vision, all while running 2 screens on windows on the same card. People seem to think I am lying or making it up. I am really just trying to help people. I am saddened whenever I see fellow 16GB VRAMers stuck on 60k context running q3 or lower quants or running weaker models because they think 27B it is out of their reach. I really just want to help them. I get told it's impossible to fit the weights let alone the KV, or that the KV must be q1 and that I am definitely not fitting vision. The trick is llama.cpp-adaptive-kv-streaming. I love this fork and by default it only supports q8_0/q4_0 but I modified it for my own purposes to support q4_0. But you can still do q8_0/q4_0 if you lower the context a bit or reduce the batch and ubatch to 128 or drop the vision maybe and increase kv-stream-arena-mib. Alternately ask you favorite LLM to add q4_0 support, it's just a few lines of code to a few files. My MTP bench normally gets around 80t/s but is slower due to the video capture, which also uses more VRAM. I did not run llama-benchy at full context in the video as it takes a few minutes, but at 262k I get around 20t/s and pp 400t/s. But the speed drop is linear and normally I compact at around 180-200k in pi anyway. But for many even those speeds are considered usable as I have seen many people satisfied with just 10-20t/s. cache-ram you can set to whatever you like it does not matter so ignore my 20GB setting. The only caveat is watch how many apps you have open that consume VRAM. For me a browser with a few tab, notepad++ and pi is enough. But nothing preventing you from dropping to the next lower quant if you like a little more breathing room. Windows will push active VRAM into shared VRAM when it runs out and your speeds will tank if you open too many apps. Some will also argue that in my video my shared VRAM goes up. That's normal as it's the KV being push to system RAM which is then streamed to the kv-stream-arena (configured to 1.5GB). For those interested I just set batch sizes to 128 and set q8_0/q4_0 KV and it slows pp by about 100t/s at high and low context on llama-benchy. Come to think of it I might just go with this KV setting myself and discard my own fork. The fork developer does not test on windows so sometimes there are compiler/linker errors but easily fixed by pointing an LLM at it. As for the frontend, don't worry about that, that's just my launcher I vibe coded. llama.cpp-adaptive-kv-streaming is command line just like llama.cpp Hope this helps at least one person.   submitted by   /u/Fancy-Snow7 [link]   [comments]