Claude Fable 5.1 just won a new coding benchmark despite failing more than six out of 10 times. Its 38.8% score comes from Real-SWE, a benchmark from Y Combinator-backed Specific Labs that takes a different approach to testing coding agents. Instead of giving them problems pulled from public repositories, it drops them into private codebases from real companies and asks them to tackle problems similar to those engineers deal with every day.The code and its solutions aren’t publicly available, which makes it even less likely that they showed up in a model’s training data.Specific Labs can’t guarantee that a model has never encountered any of the code, but the company says using private code makes that much less likely. It also estimates that 99% of tokens in real-world enterprises are hidden from frontier models. Once the agents were dropped into unfamiliar territory, the scores fell fast.The code and its solutions aren’t publicly available, which makes it even less likely that they showed up in a model’s training data.Private code changes the testFable 5.1, running via Claude Code, led the pack at 38.8%. GPT-6 Astra on Codex CLI followed at 33.8%, with Gemini 3.8 Flash on Gemini CLI at 31.2%.After that, the scores dropped significantly. GLM 5.3 scored 28.8%, Grok 4.6 and Muse Spark 1.3 tied at 23.8%, Kimi K3 hit 18.8%, and GPT-5.6 Sol finished at 16.2%.Each model got eight tries at every task. Real-SWE also tested each model with its own coding tool — Fable 5.1 with Claude Code, Astra with Codex CLI, and Gemini 3.8 Flash with Gemini CLI — so the scores reflect the full setup (not just the model).As GPT-6 Astra’s ARC-AGI score showed, changing the scaffolding around a model can change its performance. Fable 5.1 in Cursor, for example, could produce a very different result.Six tasks stumped everyoneReal-SWE uses proprietary code licensed from real businesses, including a consumer product with more than 200,000 users and a fintech platform that has processed more than 100,000 bank statements. The work also spreads across the codebase, with Real-SWE solutions touching a median of 11 files, nearly double the six-file median in benchmarks like FrontierCode and DeepSWE.On individual tasks, the scores fell even further, with six of the 10 posting success rates below 15%. On individual tasks, the scores fell even further, with six of the 10 posting success rates below 15%. A billing schedule migration had a 14.1% fix rate, API token metering landed at 12.5%, S3 storage tracking hit 10.9% and a linearizable scan came in at 4.7%, while a tax jurisdiction bug was patched just 3.1% of the time.Not a single model solved the analytics stream reducer across 64 attempts. Astra and Gemini, meanwhile, went eight for eight on a multi-region sweep and Fable solved seven of eight, yet all three failed every attempt at the linearizable scan. No agent was consistently reliable across the benchmark.Not a single model solved the analytics stream reducer across 64 attempts.Where the agents broke downFable 5.1 most often missed requirements (36.7%) or ran into integration errors (34.7%). Astra’s failures were split between integration errors and unverified assumptions, both at 34%.Integration errors appeared in nearly half of Gemini 3.8 Flash’s failed runs, while GPT-5.6 Sol made unverified assumptions in 43.3% of its failures.What 38.8% really meansReal-SWE doesn’t prove that public coding benchmarks are inflated by data contamination, and 10 tasks is still a small sample.But the top-performing agent still failed more than 60% of the time on private code it likely hadn’t seen before, suggesting that solving a coding problem is very different from finding your way through an unfamiliar production codebase.The post AI’s best coding agent fails 60% of the time — and the data backs it up appeared first on The New Stack.