AlphaEval: Evaluating Agents in Production

The rapid deployment of AI agents in commercial settings has outpaced the development of evaluation methodologies that reflect production realities. Existing benchmarks measure agent capabilities through retrospectively curated tasks with well-specified requirements and deterministic metrics---conditions that diverge fundamentally from production environments where requirements contain implicit constraints, inputs are heterogeneous multi-modal documents with information fragmented across sources, tasks demand undeclared domain expertise, and success is judged by domain experts. We present \modelname, a production-grounded benchmark of \textbf{94} tasks sourced from \textbf{seven} companies deploying AI agents in their core business, spanning six O*NET\footnote{\url{https://www.onetonline.org/find/descriptor/browse/2.A}} occupational domains. Unlike model-centric benchmarks, \modelname evaluates \textit{complete agent products}---Claude Code, Cursor, Codex, and GitHub Copilot---as commercial systems, capturing performance variations invisible to model-level evaluation. Our evaluation framework covers multiple paradigms (LLM-as-Judge, reference-driven metrics, formal verification, rubric-based assessment, and automated UI testing), with individual domains composing multiple paradigms. Beyond the benchmark itself, we contribute a \textit{requirement-to-benchmark construction framework}---a systematic methodology that transforms authentic production requirements into executable evaluation tasks in minimal time. Given any real-world production requirement, this framework enables rapid construction of fully automated, reproducible evaluations---making production-level agent assessment as accessible as research benchmarking.