There's a lot of talk here about not chasing tok/s by lobotomizing the model, and most of what I post is still speed numbers. So this is the other half: what the models I run at home actually get done on my own work, with Claude models added as fixed reference points. Everything local runs on my own engine (capy-flash) on 2x RTX 5070 Ti 16 GB, Ryzen 9900X, 96 GB DDR5. No expert skipping for flash next. The tasks I took 9 real regressions from the private history of that engine (C++/CUDA). For each one I checked out the code right before the fix and wrote the bug report the way I'd file it to myself: symptom, numbers, logs, no hint about where to look. Performance cliffs, numeric drift, a bug that only shows up on WSL2 with two GPUs. None of it has ever been public. The agent has to find the cause and write a patch. It runs sandboxed on a Mac with no network and no CUDA toolchain, so it can't build or run anything and has to reason from the code. 80 steps max, 3 runs per task per model. Claude models run in Claude Code, local models in OpenCode. I tried the local models in Claude Code through a bridge first and they did much worse there, so each side gets the harness it works best in. Scoring Every patch is scored twice: automatically, against a hidden key built from the real fix: right files and functions, patch precision, a few checks on the code; by Claude Opus 5 as a blind judge. It sees the bug report, the code, the real fix labelled as "one valid solution, not the only one" and the compiler output, and gives 1 / 0.75 / 0.5 / 0.25 / 0. A patch that doesn't compile is capped at 0.25 in both. The two scores correlate at r = 0.79 per patch and put the models in the same order. The judge is a bit stricter than the key, and it isn't kinder to Claude: judge minus auto score is negative for every Claude model and around zero for the Qwen ones. Turning a real bug into a task with a fair key is slow, and one model over 9 tasks x 3 runs takes 4 to 8 hours on my box. So 9 tasks is what I have, and the confidence intervals are wide, about ±0.2. The tiers hold. Two models less than ~0.1 apart are a coin flip. Results Judge score 0-1, tokens include reasoning. CIs and the auto score are in the image. Model Where Score Tokens/task Min/task Claude Opus 5.5 API 0.74 16k 3 Claude Sonnet 5.5 API 0.58 8k 1 Qwen3.8-Flash-Next NVFP4 local 0.50 53k 10 Qwen3.8-Flash-Next 3-bit local 0.48 53k 9 Claude Haiku 5.5 API 0.47 45k 4 Qwen3.8-27B 3-bit local 0.39 111k 18 Qwen3.8-27B NVFP4 local 0.34 81k 13 Claude Sonnet 5 API 0.28 43k 8 Claude Sonnet 4.5 API 0.19 15k 6 Claude Haiku 4.5 API 0.13 23k 4 Gemma 4 26B-A4B (thinking) local 0.07 33k 5 What 3 bits did This is the part that matters for the tok/s argument. Same models, NVFP4 vs 3-bit, paired on the same tasks. Flash-Next (3 bits on the experts only): score -0.02, tokens x1.03. As far as 9 tasks can tell, it's free. 27B: score +0.05, inside the noise. But it generates 40 % more tokens per task (x1.40, CI 1.25-1.59). The 3-bit 27B decodes faster on my engine, 153 vs 132 tok/s, and still needed 18 minutes per task instead of 13. Part of that gap was a slow 3-bit prefill path I've fixed since; a rerun on the newer build still came out at x1.30 tokens. So the damage from quantization didn't show up as a lower score. It showed up as the model thinking longer to get to the same place. Looking only at tok/s, the 3-bit 27B is the better deal. Per bug fixed, it isn't. Non-compiling patches don't tell a clear story: Flash-Next 5/27 at 3-bit vs 1/27, the 27B 2/27 vs 4/27. The rest Flash-Next at home sits with Haiku 5.5 and above the older Sonnets. Sonnet 5.5 is a step up, Opus 5.5 is ahead of everything. Tokens per task say as much as the score. Sonnet 5.5 gets further with 8k tokens than Flash-Next with 53k, which is 10 minutes per bug on my box against a minute and a half. The dense 27B is behind the MoE (Flash-Next +0.16 paired, P = 0.92) and burns more tokens getting there. Gemma 4 26B didn't work as an agent here. With thinking off it never edited a file in 9 tasks. With thinking on it edits, mostly wrong. Limits 9 tasks from one codebase in one domain (GPU inference), bug reports written in Italian. This is how models do on my work, not a general ranking. Different harnesses for Claude and for the local models. The judge is a Claude model. The automatic score is there to check it and they agree, but it's still a limit. No builds or tests, so this measures diagnosis and patching from reading code, not the full edit-build-test loop. I'm not publishing the tasks, they'd stop being useful. If you've checked your own quants on real work instead of perplexity or KL, I'd like to know whether you see the same pattern: same score, longer thinking.   submitted by   /u/FeydRowan [link]   [comments]