Measuring When Multi-Agent Collaboration Pays: A Matched Four-Protocol Trace Resource for Scientific Reasoning

Agent evaluation overwhelmingly reports final-answer accuracy on independently sampled problems. This measures whether a system succeeded but not the quantities that govern real multi-agent deployment: which collaboration protocol to run, at what cost, and whether the collaboration was worth it. We argue these are measurement questions that passive, single-run, final-accuracy benchmarks structurally cannot answer, and we release a resource designed to answer them. Every problem is solved under the same four collaboration protocols: direct baseline, single-agent, planner-executor-reviewer (PER), and broadcast deliberation, yielding counterfactual per-problem outcomes with aligned token, model-call, and process signals. We treat the matched runs as measurement instruments and demonstrate one, the oracle cheapest-successful protocol: it shows that “when collaboration pays” is strongly domain-dependent. Expensive collaboration is the cheapest successful choice for only 12.9% of competition-math problems but 28.6% of biology-science QA problems, a contrast invisible to accuracy alone, with math-slice evidence that the collaboration share is stable across two actor families. We release the matched traces, labels, schema, and documentation as a versioned, openly documented resource for measuring multi-agent collaboration.