I asked Qwen3.8 how this is done, because it works so well honestly it's like having a local claude. I have done a few shot prompts, like a multiplane flight sim etc. dam amazing even at this low quant. so here are my details per Qwen3.8 27b on how it's setup. so meta. lol FYI I use PI coder ## The rig - **GPU:** AMD Radeon RX 7600 XT, 16 GB (Navi 33 / gfx1102) - **CPU:** i7-8700 (6c/12t), 32 GB DDR4 - **Driver path:** Vulkan (RADV), not ROCm — `--device Vulkan1` because `Vulkan0` is my Intel iGPU ## The app **llama.cpp `llama-server`**, build **10480** (commit `01818e495`), Vulkan build. It's just an OpenAI-compatible API on localhost — I'm driving it from pi (a coding agent), but any OpenAI-compatible client works. Model: **unsloth `Qwen3.8-27B-UD-IQ3_XXS`** -- 10.2 GiB GGUF, natively 262k context. ## The actual command ```bash llama-server \ -m Qwen3.8-27B-UD-IQ3_XXS.gguf \ -c 163840 \ -ngl 999 --kv-offload \ -ctk q8_0 -ctv q8_0 \ -fa on \ -b 512 -ub 256 \ -t 6 -tb 12 \ --jinja --reasoning-preserve \ --reasoning-effort medium \ --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 \ --device Vulkan1 \ --no-mmproj -np 1 \ --host 127.0.0.1 --port 8080 ``` The two flags that matter for context size: -c 163840 and the small compute buffer -b 512 -ub 256. MTP / speculative decoding is off on this config. Why 160k fits in 16 GB (the part that sounds made up) This GGUF is a hybrid model, not dense attention. Only 16 of its 64 layers keep a full KV cache; the other 48 layers hold constant-size SSM state instead. So KV is only: 16 layers × 4 KV heads × (256 K + 256 V) at q8_0 ≈ 34 KB/token and 163840 × 34 KB ≈ 5.3 GiB of KV. Budget: ~10.2 GiB weights + ~5.3 GiB KV + ~0.9 GiB compute/SSM overhead ≈ ~16.4 GiB against 16 GB — so this is right at the edge, and the script's pre-flight VRAM warning is expected and fine. You're not OOMing, it's verified working. (If your decode drops to ~6 t/s instead of ~18, that's your GPU spilling to system RAM, not a crash — back off context.) Measured numbers per users config - 18 t/s decode at 160k - 141 t/s prefill on a 1.8k-token prompt For reference, the same model with MTP on: ~24 t/s at 98k ctx, ~39 t/s at 124k. So 160k is the MTP-off trade: longer context, plain decode. Honest caveats - 160k is a VRAM-math ceiling, not a quality claim — don't ask the model to read 150k tokens and remember page 1, I won't : ) - You need the GPU otherwise idle (no ComfyUI eating 4 GB in the background) - The model puts most output in reasoning_content; clients that only render content show a blank bubble — check the raw response before you blame the server   submitted by   /u/According_Study_162 [link]   [comments]