How well do agents use test and verification techniques?

Why we picked it · the editor's summary

Dan Luu had Codex with GPT-5.6 Sol write a Zstd implementation in Rust under 26 prompt conditions, about 80 runs each, scored on the share of runs that passed every hidden test. Naming a technique rarely bought its benefit: asked for Verus or Lean, agents proved abstract or arithmetic properties far from the bug-prone code and leaned on ordinary Rust tests anyway. The condition with no testing instructions scored above average. The best was Luu's own skill, which has the agent list likely subtle bugs and alternative readings of the spec before it writes code. If your agent instructions say "use property-based testing" and stop there, build the test and triage structure yourself first; Luu found agents add to one "ok-ish" without much supervision.