SOP-Bench: Can Agents Follow Standard Operating Procedures Across Long-Horizon Tasks?

Real-world work runs on standard operating procedures (SOPs): long, multi-section documents that tell an employee who may authorize a termination, when an invoice needs a second signature, and a hundred other rules they must apply to everything they touch across a shift of separate decisions. We increasingly deploy language-model agents the same way, placing a long SOP in context and trusting it to govern the agent's behavior across an entire multi-step job. SOP-Bench tests whether that trust holds, and finds it largely misplaced. No frontier model completes even a quarter of its tasks; along the way, agents fire employees without authorization, clear an expense the submitter approved himself, and forward expired medical records to insurers. The benchmark casts each task as a test of context-time memory (whether information supplied at inference stays in force across a long horizon of actions) as opposed to weight-time memory baked into parameters. Each of its 100 tasks places an agent in a self-contained enterprise world (files, email, Slack, Jira, and a calendar behind MCP servers) governed by a realistic SOP that averages 43 pages and 22K tokens and runs as long as 124 pages. Because each task uses a modified SOP, a model cannot fall back on memorized knowledge and must read the document in front of it; grading is deterministic and scores prohibited actions as well as required ones. We trace the failures to four recurring ways an agent loses its grip on a long standing instruction over a long horizon, a regime we argue is undertested by existing long-context and agentic benchmarks.