Coding agents are good at writing a test next to a fix. Whether that test would have caught the bug is another question. I kept seeing tests that pass just as well on the old, broken code. So I built Receipts. It runs every test the change added or edited twice: once with the change, once with the source files put back the way they were before it. A test that passes both times proves nothing about the fix (THEATER). One that fails on the old code and passes on the new one is PROVEN. There's no LLM in the check, it's plain git plus your own test runner. Then I ran it on 181 real changes from 17 open-source repos: maintainer fix commits vs PRs written by Claude Code, Codex, Copilot and Cursor. Most changes on both sides came with a test that fails without the change (93% vs 88%), so agents aren't as bad as people say. The difference was how they fail. In 10% of agent PRs every test failed on the old code only because the test file imports a name the change adds, so on the old code the file can't even load. None of the maintainer commits looked like that. This week someone on r/ClaudeAI pointed out that "it went red" isn't enough proof, so now every PROVEN test also shows how it failed on the old code, e.g. "on the old code: assert not True". That way you can see it actually reached its assertion. How I built it: TypeScript, no runtime dependencies, written with Claude Code over a few weeks. The hard part wasn't the idea, it was the edge cases from real repos: parametrize tables where a new case is one row, timezone-bound suites, tests that import an installed copy instead of the checkout. Works with pytest, vitest and jest. CLI, GitHub Action, and a Claude Code plugin (it's in Anthropic's plugin directory as Receipts Check). Free, MIT. Repo: https://github.com/syntaxixr/receipts The study, every change linked: https://github.com/syntaxixr/receipts/blob/main/docs/study.md Feedback welcome, especially on cases where you think the verdict would be wrong.   submitted by   /u/Sanechka_SS [link]   [comments]