I am wide open to the idea that I am misguided or misinformed, but my findings feel worth sharing: Short video of my UI with 8 agents each explaining a codebase in parallel long prose: https://imgur.com/JSqlXIi A few days ago I read the Stanford/Berkeley TokenGate paper. The idea is to make long context cheap by pruning tokens up front. I tried it. It's fine for one question, but agents keep asking new ones, and by then the tokens they need are gone, so code recall drops hard. So I went the other way - don't throw anything away, just don't look at all of it at once. Keep the whole context in system RAM and at each step pull in only the ~1% that matters right now. A few days later, I have two 3090s run 8 coding agents at once, each holding a 250K-token context: Every decode step, each attention layer reads only ~1% of the tokens, picked with a tiny rank-64 "key sketch". The full KV cache lives in system RAM (int4). The GPU keeps the sketch and a small hot set. Only the picked tokens cross PCIe. What it took: - Triton kernels that build the sketch, score every token, pick the top ~2K and pack them for the stock int8 attention kernel. CUDA-graph safe. - vLLM patches that switch dense/sparse per step, so short chats stay full speed. - A RAM tier for the KV cache with a prefix cache and recurrent-state checkpoints, so follow-ups don't recompute. - Follow-ups fetch only the picked rows. v1 re-copied the whole document every turn (~8 GB), which capped me at 2 agents per card. - A scheduler fix so one agent reading a huge prompt doesn't freeze the rest. - A frozen sketch basis (one file) plus the script to refit it. It's all in the repo as patches, not a fork. Apply them on top of HyperQwen @ ce9615c (a public vLLM fork) and run the install script. Tests and every bench script and result are included. Model: dbirks/Qwen3.8-27B-W4A16-AutoRound + HyperQwen's prepare/fetch_fast_variant.py. All numbers were measured with gpu-unlock loaded (a kernel module that roughly doubles 3090 BF16 tensor throughput, mostly faster prompt reading). Measured (Qwen3.8-27B W4A16, MTP-7, 3090 Ti + 3090 at 250 W): - 8 × 250K: ~545 tok/s total on explain-style output, ~790 on copy-heavy. - 8 agents at 176–205K generating nonstop for 6 min: 657 tok/s box average, 69–90 per agent. - Recall at 250K: 31/32. Follow-ups start in ~5 s. - Single long chats are faster too: +15–41% at ~113K vs dense. Caveat: attention is approximate, so near-duplicate code lines occasionally get mixed up. Code: - Serving (patches + kernels): https://github.com/bradj00/sparsekv-hyperqwen - Experiments: https://github.com/bradj00/sparsekv-lab Rig: * i9-12900K * Z790 motherboard * 96 GB DDR5 RAM * 3090 Ti + 3090, power capped at 250 W   submitted by   /u/o2beast [link]   [comments]