We ran DeepSWE, Terminal Bench, Automation Bench, HLE, SciCode at 512K context on our DeepSeek UD version 3.1 GGUFs and show we're >10-20% better than GSQ-RCO We ran all other benchmarks (14 in total) to 128K. AIME2025 64x, DeepSWE/TB 3x. We addressed some issues people mentioned with these new benchmarks: 32K context was too short - we did 512K context for agentic + 128K for general tasks Saturated benchmarks - new unsaturated agentic style ones including Terminal Bench, DeepSWE and Automation Bench Benchmarks are noisy - we did 35.1B tokens of benchmarking vs 1.3B before so 27x more data. See this section where we did 64x AIME 2025 runs, 3x DeepSWE, 3x TB runs and more. Our graders are wrong - we used Math Verify, improved graders, used gpt-6-luna xhigh for HLE, and all official graders for Terminal Bench (Harbor), DeepSWE and the rest. See graders and infra fixes For DeepSWE v1.1, we used mini-swe-agent and chose 500 max steps (334 steps was the max runs we saw, so none went over). Terminal Bench 2.1 used mini-swe-agent as well, and we set max steps to 150 with no max limit. We had to exclude 7/89 tasks since they couldn't pass on our infra - see https://unsloth.ai/docs/models/deepseek-v4.1-flash#excluded-terminal-bench-tasks-due-to-issues We also re-checked ISTA's numbers as we have done many more runs (27x more data). GSQ-RCO IQ1_M claims 98.7% recovery, but we measured 97.24 [96.23, 98.21] with a p-value of 0.0013. GSQ-RCO IQ2_XS retaining 100% claimed recovery is possible with a p-value of 0.21. We measured 98.38% [97.52, 99.22]. So this is possible. GSQ-RCO IQ2_XXS claims 99.80% recovery but we measured 98.38 [97.52, 99.22] with a p-value of 0.0001. So this is nearly impossible. Paired difference to the official base (our runs, same questions, sign-flip permutation test, two-sided): AIME25 GPQA-D LCB v6 GSQ-RCO IQ1_M -1.61 (p=0.02) -2.89 (p=0.004) -3.24 (p=0.00002) GSQ-RCO IQ2_XXS -0.47 (p=0.25) -2.07 (p=0.02) -2.00 (p=0.01) GSQ-RCO IQ2_XS -0.42 (p=0.48) -0.51 (p=0.42) -0.38 (p=0.52) The claimed recovery tested directly against our measurement (the card's definition: sum of the 3 scores / the base's sum, vs official FP8/FP4): p (no loss) = paired sign-flip permutation test of recovery = 100%; p (recovery >= claim) = one-sided paired bootstrap over questions. Claimed recovery Measured recovery [95% CI] p (no loss) p (recovery >= claim) GSQ-RCO IQ1_M 98.70 97.24 [96.23, 98.21] 0.00002 0.0013 GSQ-RCO IQ2_XXS 100.00 98.38 [97.52, 99.22] 0.0003 0.0001 GSQ-RCO IQ2_XS 99.80 99.54 [98.88, 100.18] 0.16 0.21 If we can get more details on how the benchmarks were run from GSQ-RCO, we are more than happy to run more. We provided all our setups and methods in our blog at https://unsloth.ai/docs/models/deepseek-v4.1-flash#dynamic-v3.1-update-november-2nd-2026 All details and data and methods at https://unsloth.ai/docs/models/deepseek-v4.1-flash#dynamic-v3.1-update-november-2nd-2026. Our new Unsloth Dynamic 3.1 GGUFs for DeepSeek are at https://huggingface.co/unsloth/DeepSeek-V4.1-Flash-1017-GGUF   submitted by   /u/danielhanchen [link]   [comments]