I've been collecting GitHub issues where an agent reported success that didn't happen. Some real patterns: - "Tests pass," but the test runner reported a failure - "Installed," but no file was written - "Restart succeeded," but the process was dead - "Message sent," but the channel didn't exist - A subagent with zero tool calls claiming it made an artifact The common thread: the agent's report and the tool's return value are treated as evidence. They aren't. I built a small Python library to test the opposite approach. It ignores what the agent says, checks independent evidence (re-runs the test, reads the actual record), and returns one of: - MISMATCH: the evidence contradicts the claim - VERIFIED: the evidence matches the claim - UNVERIFIABLE: no evidence, so it doesn't guess The video shows it on a coding agent: the claims are scripted, the test runs are real. Honest limits: it only checks claims against evidence you define. It doesn't prove the code is correct, and claim detection uses English heuristics, so it can miss things. pip install cogext-observe Curious how others handle this. Do you verify agent "done" claims at all, or just review diffs?   submitted by   /u/xspyyy [link]   [comments]