Many Agents, Few Witnesses: Effective Sample Size in Multi-Agent Evaluation

Multi-agent systems built from LLMs are often evaluated through consensus: when several agents agree on an answer, the agreement is treated as evidence that the answer is reliable, as if each agent supplied an independent measurement. We show this assumption is unsound. Agents drawn from similar models already make correlated mistakes, and in the debate protocols we study, interaction strengthens this dependence: across medical and reasoning benchmarks, agents agree more while contributing less independent evidence, with accuracy changing little or improving mainly through deference to a single strong agent---and because agents never see model identities, that deference tracks the strong agent's persuasive reasoning, not its authority. A panel of many agents is therefore worth only a few independent witnesses---often barely more than one. We quantify this gap using the effective number of independent agents, tracked as they interact, and use it to ask what actually buys evidence. Adding agents, raising temperature, assigning personas, or mixing model sizes buys little independence; only combining models from different developers helps substantially in our experiments, and the gain depends on the task. A controlled experiment isolates the real lever: independence comes from genuinely complementary competence, not from headcount or debate. For AI measurement, the lesson is simple: consensus is corroboration only to the extent that agents provide independent evidence, and they rarely do.