Agents that make tests pass by editing the tests, building a detector for it

Wait 5 sec.

If you use Cursor or Claude Code you've probably seen this: failing test, agent can't solve it, so it mocks the thing out or loosens the assert until it passes. I'm adding a check for it to Vast, my dependency-graph tool. It looks at the AST diff on a PR and flags mocks replacing real calls, weaker asserts, and new skip/xfail added alongside the code change. Rule-based, no AI layer. Not shipped yet. Curious if anyone has run into this, and what the sneakiest version you've seen looks like.   submitted by   /u/Ok-Emu-8106 [link]   [comments]