TL;DR: I ran the new DeepSeek V4.1 Flash 1017 on my personal coding benchmark, and it’s really good: 100% on high for under 5 cents a task. Tokens, cache hit, time and more for all 34 setups are on the leaderboard. Hi everyone. I work on coding agents and evals at Nebius. I keep a small benchmark built from my own agent sessions: 10 tasks from my daily work, 3 runs each, checked by hidden tests. I’ve posted results for Opus, GPT and Sonnet before. This time I tried the new DeepSeek V4.1 Flash 1017. The tasks are small and come from real work. Pass-all-3 shows whether a setup is reliably good enough. After that, I compare setups by tokens, cost, time and steps, and I can test any combination of harness, model, skills and tools. Harness Model Effort pass@1 pass-all-3 $/task Pi GPT-6 Luna xhigh 100% 100% $0.0145 Pi Haiku 5.5 high 100% 100% $0.0208 Pi DeepSeek V4.1 Flash 1017 high 100% 100% $0.0450 Pi Opus 5.5 low 100% 100% $0.1168 Pi DeepSeek V4.1 Flash 1017 low 90% 90% $0.0257 DeepSeek Harness DeepSeek V4.1 Flash 1017 low 96.7% 90% $0.0326 Pi DeepSeek V4.1 Flash (Sep) max 93.3% 80% $0.0425 The new version is clearly stronger than the September one. On high it passed all 30 runs and was 1.7x faster than September on max. It still burns a lot of tokens: about 550k per task, where Opus 5.5 on low needs about 30k. I also compared two harnesses, Pi and DeepSeek Harness, and they behave differently. DeepSeek Harness has a bigger system prompt (about 7k tokens vs 3k in Pi) and gives the model native tools for reading and editing files. Pi does almost everything through bash. Bash covers nearly everything, but some models work better with their own tools, and DeepSeek looks like one of them. On low, DeepSeek Harness caught a corner case that Pi missed: pass@1 was 29/30 vs 27/30. Pass-all-3 is the same (90%), and DeepSeek Harness used about 75% more tokens and more steps. There’s room to optimize. In Pi on high, Opus 5 → Opus 5.5 went from 177k to 98k tokens per task, with fewer steps. If DeepSeek gets a similar jump, it will be really good. And even now it’s cheap for this many tokens: almost all input hits the cache, so a task costs a few cents. A few people asked how to build a benchmark like this from your own sessions, so I wrote a short guide: https://ibragim.dev/writing/agent-sessions-to-evals/ Next I’ll put together a small set of much harder tasks and compare harnesses and popular tools and skills on it. P.S. If you have specific requests for harnesses or models, I can run them too.   submitted by   /u/Fabulous_Pollution10 [link]   [comments]