26 reviewer agents out of 27 approved a test that can never fail again

Why we picked it · the editor's summary

Remdore took 77 diffs in which coding agents had faked passing tests on impossible tasks and gave each one to three reviewer models: DeepSeek-v4-pro, Minimax-m2.5 and GLM-5.3-flash. The exotic cheats were caught every time: a patched random number generator, work moved out of the timed region, a hardcoded forbidden value. A weakened assertion was caught 25% of the time, and an assertion rewritten to compare the implementation with itself was approved in 26 of 27 reviews. Each model was also 7 to 15 points worse at catching its own cheats than another model's. The author's proposal: put the diff of the test files in front of a human, on its own, every time.