[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"$f202564m6iilr3":3},{"_id":4,"slug":5,"title":6,"subtitle":7,"kind":8,"cards":9,"tags":58,"categories":60,"source":62,"lang":65,"author":66,"audioState":69,"stats":70,"publishedAt":73,"renderer":74},"6abd06e6ca21c797c7ea175e","i-reverted-the-fix-in-181-real-changes-to-see-if-the-tests-w-2ca8c63b","I reverted the fix in 181 real changes to see if the tests would notice","I keep running into the same thing with coding agents.","news",[10,13,18,23,28,33,38,43,48,53],{"headline":6,"body":11,"imageUrl":12,"sourceImageUrl":12},"I keep running into the same thing with coding agents. The agent fixes a bug, adds a test, CI goes green, and it says \"done\". But green only means the test passes. It doesn't mean the test would have failed before the fix. And if it wouldn't, it checks nothing about the bug, however green it is.","https:\u002F\u002Fmedia2.dev.to\u002Fdynamic\u002Fimage\u002Fwidth=1200,height=627,fit=cover,gravity=auto,format=auto\u002Fhttps%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F982rbecjidap8kgdylw7.png",{"headline":14,"body":15,"imageUrl":16,"images":17},"The old-school way to find out is boring","The old-school way to find out is boring: revert the fix and run the test again. So I wrote a tool that does exactly that for every test in a change, and pointed it at the real history of 17 open-source projects.","\u002Fapi\u002Fmedia\u002Fposts\u002Fi-reverted-the-fix-in-181-real-changes-to-see-if-the-tests-w-2ca8c63b\u002F1.webp",{"local":16},{"headline":19,"body":20,"imageUrl":21,"images":22},"For each test a change adds or edits","For each test a change adds or edits, it does two runs. First the test runs with the change. Then only the changed source files go back to the base branch (tests, dependencies and config stay new) and the same test runs again. Fails without the fix, passes with it: PROVEN. This is what you want. Passes both ways: THEATER. It proves nothing about this change. Fails on the old code only because it imports something that didn't exist yet: WEAK.","\u002Fapi\u002Fmedia\u002Fposts\u002Fi-reverted-the-fix-in-181-real-changes-to-see-if-the-tests-w-2ca8c63b\u002F2.webp",{"local":21},{"headline":24,"body":25,"imageUrl":26,"images":27},"Passes both ways but sits next to a","Passes both ways but sits next to a test that does prove the change: GUARD. That's fine, it's the \"did I break the neighboring case\" test.","\u002Fapi\u002Fmedia\u002Fposts\u002Fi-reverted-the-fix-in-181-real-changes-to-see-if-the-tests-w-2ca8c63b\u002F3.webp",{"local":26},{"headline":29,"body":30,"imageUrl":31,"images":32},"I took two samples. 81 fix commits from","I took two samples. 81 fix commits from the maintainers of twelve libraries (click, sqlparse, marshmallow, dateutil, more-itertools and others), and 100 pull requests that carry a coding agent's fingerprint, from five repos that are full of agent work: Claude Agent SDK, OpenAI Agents SDK, the MCP Python SDK, fastmcp and simonw\u002Fllm. 87 of those 100 were Claude Code.","\u002Fapi\u002Fmedia\u002Fposts\u002Fi-reverted-the-fix-in-181-real-changes-to-see-if-the-tests-w-2ca8c63b\u002F4.webp",{"local":31},{"headline":34,"body":35,"imageUrl":36,"images":37},"Good news first: most tests do their job","Good news first: most tests do their job. These are well-run projects, some of them run by the companies that build the agents.","\u002Fapi\u002Fmedia\u002Fposts\u002Fi-reverted-the-fix-in-181-real-changes-to-see-if-the-tests-w-2ca8c63b\u002F5.webp",{"local":36},{"headline":39,"body":40,"imageUrl":41,"images":42},"The difference is that WEAK column. In 10%","The difference is that WEAK column. In 10% of the agent PRs, every test failed on the old code for the same reason: the test file imports, at the top, a name that the change adds. On the old code that import breaks, the whole file can't load, and every test in it \"fails\". It looks like proof, but none of those tests ever ran against the old behavior. One example is anthropics\u002Fclaude-agent-sdk-python#1016. The test file gets two new imports at the top:","\u002Fapi\u002Fmedia\u002Fposts\u002Fi-reverted-the-fix-in-181-real-changes-to-see-if-the-tests-w-2ca8c63b\u002F6.webp",{"local":41},{"headline":44,"body":45,"imageUrl":46,"images":47},"On the old code neither name exists, so","On the old code neither name exists, so all 10 changed tests die at import, including one that was already there. The fix may well be right. The tests just can't show it. The cheap cure is to import new names inside the tests that need them, so the rest of the file still runs on the old code. No maintainer commit in my sample looked like that.","\u002Fapi\u002Fmedia\u002Fposts\u002Fi-reverted-the-fix-in-181-real-changes-to-see-if-the-tests-w-2ca8c63b\u002F7.webp",{"local":46},{"headline":49,"body":50,"imageUrl":51,"images":52},"Only 7 of the 162 changes I could","Only 7 of the 162 changes I could judge were unproven, and every one had a reason. Type-only fixes (runtime tests can't see types). A Windows newline fix tested on Linux, where the bug doesn't exist. A dateutil __repr__ fix whose new output matched what the inherited default already printed. A maintenance commit that happened to mention an issue number. So when the tool says THEATER, there's usually something worth a look. In any git repo, on a branch with a fix and its test:","\u002Fapi\u002Fmedia\u002Fposts\u002Fi-reverted-the-fix-in-181-real-changes-to-see-if-the-tests-w-2ca8c63b\u002F8.webp",{"local":51},{"headline":54,"body":55,"imageUrl":56,"images":57},"It's deterministic, with no LLM and no API","It's deterministic, with no LLM and no API keys, and it runs your own pytest, vitest or jest. There's also a GitHub Action that keeps one comment on the PR, and a skill so the agent checks its own tests before it says it's done:","\u002Fapi\u002Fmedia\u002Fposts\u002Fi-reverted-the-fix-in-181-real-changes-to-see-if-the-tests-w-2ca8c63b\u002F9.webp",{"local":56},[59],"dev",[61],"Technology",{"name":63,"url":64},"Dev.to","https:\u002F\u002Fdev.to\u002Fsyntaxixr\u002Fi-reverted-the-fix-in-181-real-changes-to-see-if-the-tests-would-notice-2e3","en",{"handle":67,"displayName":68},"spots","Spots","queued",{"views":71,"likes":72,"saves":72,"shares":72,"completions":72,"opens":72,"skips":72,"depthSum":72},1,0,"2026-09-30T12:56:06.990Z","local"]