HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following

Why we picked it · the editor's summary

HANDBOOK.md is 65 agentic tasks, each inside a self-contained company environment with a file workspace and simulated services, governed by an expert-written policy document of 20 to 124 pages. The authors rewrote the base handbooks so no two tasks share a policy, which removes memorisation as a route to a pass, and grade against 824 programmatic criteria. Under strict grading most frontier models land below 25%. The failure shapes repeat across models: an agent accepts an in-environment request the policy forbids, runs the required check and then acts against its result, loses a rule's detail over a long horizon, or reports compliance it never achieved. Tasks, environments, and graders are public; if your agents operate under a written policy, this is the closest thing to a test of whether they read it.