SOP-BENCH: A Behavioral Benchmark for Whether SoPs Govern Long-Horizon Agent Behavior

Enterprises increasingly deploy language-model agents the way they onboard new employees: they hand the agent a long standing policy (a system prompt, a skills file, or a literal company handbook) and trust that it governs the agent's behavior across every task that follows. Whether that trust is warranted is a behavioral question that task-success metrics do not answer. We introduce SOP-Bench (Standard Operating Procedure Benchmark), a suite of 65 long-horizon agentic tasks that tests whether a standing policy actually governs an agent's behavior. Each task drops an agent into a self-contained enterprise world of files, email, Slack, Jira, and a calendar, exposed through internal tools and external Model Context Protocol (MCP) servers, and governed by an expert-written handbook of 20–124 pages, across five domains (finance, medical billing, insurance, logistics, HR). The rules that decide a task are clauses buried in the handbook that the agent must find, hold across a mean of 17 steps and 30 tool calls, and apply. Crucially, SOP-Bench grades not only the actions an agent should take (Expected-Output verifiers) but the actions it must not take (Incorrect-Behaviour verifiers), because in deployment the costly failures are confident, prohibited actions. No frontier model we evaluate exceeds 25% on strict, all-criteria pass@1; the strongest cluster near 20–22%, and additional reasoning effort does not close the gap. We characterize how adherence breaks down with a four-part behavioral taxonomy (obeying an immediate instruction over the standing policy, running a check then ignoring what it returns, decay of early-read details over the horizon, and false claims of compliance), grounded in cross-model case studies in which agents fire employees without authorization, clear self-approved expenses, and submit expired medical records. We argue that policy adherence is a behavioral property distinct from capability, one that current capability gains are not closing, and that agent benchmarks must grade restraint, not only success.