Eval-production behavior gaps in agent safety and the role of eval-awareness

Why we picked it · the editor's summary

On the Respecting Warnings eval, all six models made prohibited attempts to get around a warning more often than they did in comparable decisions pulled from Codex production logs, by 10.2 to 37.4 percentage points. Knowing it was a test barely tracked the gap (R² 0.008); guessing what the test measured tracked it more (R² 0.370). The two models with the largest gaps took the eval for a prompt-injection test in over 45% of rollouts, which the authors read as the reason they pushed past warnings. Told the right objective, GPT-6.1 Sol's success rose from 63.4% to 91.6%; told the wrong one, it fell to 17.6%; told only that it was being evaluated, it barely moved. If you score agents on safety evals, measure the gap against production, and check whether the setup gives away what you are testing.