I'll be honest, I don't know shit about how models work under the hood, but I used to work on idle/incremental/clicker games professionally, and I really like watching numbers go up. And since I'm relatively GPU poor compared to some of you, I was excited to hear about Raon and figured it'd be a fun project to vibe on, and to see if I could get it to fit in VRAM. I only have a 5070 Ti with 16GB VRAM, a Ryzen 5 5600X, and 48GB DDR4. I first tried KRAFTON's Q4_K_M with 9 layers' experts in RAM (best fit) and I got 50 tok/s. That's when I became obsessed with how far I could squeeze it and after 4 days and 40% of my Claude Max 20 plan, decode speed went from 50 tok/s to 232 tok/s at roughly the same quality as the Q4_K_M. GGUF: McNiiby/Raon-1-mini-preview-IQ3_KS-GGUF · Fork with build/run instructions: Mcniiby/ik_llama.cpp-raon (needs a 16 GB card) The first win was the quant. On regular llama.cpp, a 3-bit quant that fit fully in VRAM jumped from 50 to ~135 tok/s. I then moved to ik_llama.cpp for the same quality in less space. Out of the box, ik actually ran Raon slower than regular llama.cpp (~106 tok/s). I'd have preferred 4-bit since ik's IQ4_KS is noticeably closer to the Q4_K_M I started with (KLD 0.05 vs 0.09 for IQ3), but at 16.3GB it didn't fit. I tried pruning experts or narrowing them, and every GB saved cost more quality than IQ4 bought back. Putting a few layers' experts in system RAM instead worked, but would've capped speed around 130 tok/s. So IQ3, fully on the GPU, won. (lower KLD means the output probability distribution is closer to the Q4_K_M, but it does not directly measure task accuracy.) (lower KLD means the output probability distribution is closer to the Q4_K_M, but it does not directly measure task accuracy.) I also trimmed the vocabulary. Raon has 260K tokens, many of them for Korean, Chinese and Japanese, and I only use English. Dropping 54K cost almost nothing (KLD 0.0931 vs 0.0923). It also exposed a sampler that still scanned the full list every token; fixing that cut sampling from 1.3 ms to 0.3 ms per token, about 15% faster in real chat. So why was ik slower than llama.cpp? The quant wasn't the problem. On a shared test file, regular llama.cpp ran 133 tok/s and ik 106, and the GPU was using only about 20% of its memory bandwidth. ik's fast path didn't support Raon's SITU activation, which switched off CUDA graphs, and Raon's mHC layers ran as ~1,500 tiny kernel launches per token. Fixing those two took ik to 147, past regular llama.cpp. After that it was a hunt for overhead anywhere in the token. Each speedup below is an A/B on my chat model within one run: Change What it fixed Speedup when added KLD vs without it SITU + mHC fusion CUDA graphs back on; mHC from ~1,500 launches per token to ~150 +31% 0.0044 Small-matmul kernel 146 small F16/F32 matmuls per token were going through cuBLAS +12% 0.0046 Fused delta attention Raon's recurrent-layer update in one kernel +9% 0.0026 MoE megakernel One persistent kernel for each layer's expert block +7% 0.0036, greedy text identical Dense matvec merge, small fusions Projections that share an input launch together; fewer tiny kernels +13% 0.0028, greedy text identical Pipelined megakernel Loads the next expert while computing the current one +5% 0.0032, greedy text identical Faster mHC boundary A latency-bound kernel that runs 49 times per token, cut from 6.5 to 3.7 µs per launch +3.1% 0.0037 Small-kernel fixes Layer-0 FFN and recurrent-layer matvecs, one fewer launch per layer +1.2% 0.0040, greedy text identical Long-context attention Past 4K tokens, decode attention splits over all 70 SMs instead of 64 +2.3-3% at 16K context 0.0006 Prompt reading (KDA scan) ik's delta-net scan was ~10× slower per step than mainline's +35% on a 4K prompt 0.0045 Each row is measured with the rows above it already on, so they multiply rather than add. So together that got me to 232 tok/s on the chat model, with weights streaming at about 2/3 of the card's peak bandwidth. I also tried KRAFTON's DFlash drafter, but fitting it meant giving up context to free VRAM and it wasn't worth it. It ran 36-53% slower, because checking a batch of 8 guesses costs 2.2× one token and only ~2.3 got accepted. N-gram speculation, which is much cheaper, added about 6% in chat, and 12-22% when the reply copies from the prompt. So does it still think well? That depends if you think Raon thinks well to begin with haha, but yes mostly it stayed the same as the base model. GPQA and AA-Omniscience came in lower, and I couldn't figure out why. These are single runs, mostly on subsets, so anything within a few points is noise. The ones above the card don't mean it's better than the original. Benchmark (thinking on, card settings) Ours Card IFBench (prompt, loose) 84.7 84.2 MultiChallenge 58.9 59.0 MMLU-Pro (210 items) 78.6 76.5 AIME 2026 94.2 93.3 GPQA Diamond 71.2 84.3 AA-Omniscience -3.0 +3.5 I also ran it on some of the evals I built for Workybara.ai to see how it does outside the usual benchmarks. At pulling structured data out of documents, it extracted better content than the small models I fine-tuned for the job, but 14% of the time it got stuck in a loop listing variations of the same item ("Deployment Promise, Deployment Vow…") until it ran out of tokens. I tried it on philosophy questions judged by Claude and it hit about 70% of the key points, but 59% of its answers had at least one clearly false claim. KRAFTON's own playground (the full model, not my quant) showed the same habits when I asked it for jokes: it broke the rules of its own password joke, called jokes it had just made up "classics," said "nothing I fetched supports it" without fetching anything, and agreed with every criticism I made, even the wrong ones. None of that shows up in standard benchmarks. I was hoping to use this to replace frontier chatbots, but even with search and RAG Raon really couldn't compete with them, and it was frustrating to chat with. How it was built Claude Code and its sub-agents did essentially all of the research, code and measurement, and helped me write this up. Over four days I typed about 15,000 words into Claude, mostly questions. My longest message was 124 words, and it was me telling it that it had jumped to a conclusion. I did rent some cloud GPUs to run the public benchmarks, so my 5070 Ti could keep doing speed work. Other than that, my part was just being the annoying kid that kept asking questions, pushing back, and saying you can do it buddy. The hunt for a good MoE to chat with that fits in VRAM will continue. I'm likely going to see if there's any more juice to squeeze on Gemma-4-26B-A4B or Qwen3.6-35B-A3B next.   submitted by   /u/McNiiby [link]   [comments]