Rethinking the Evaluation of Harness Evolution for Agents

We revisit the evaluation of automatic harness design for LLM agents. Recent methods have reported promising gains by framing harness design as a form of agent self-evolution, in which a meta agent searches over prompts, tools, memory, and scaffolding. However, many existing harness evolution methods use unit test cases to search for harness configurations and then report final performance on the same public benchmark. This protocol raises two fundamental concerns. First, because the search process already exploits test-case feedback, it is unclear whether harness evolution provides gains beyond simple test-time discovery baselines. Second, evaluating on the same benchmark used for search introduces a risk of overfitting. To address these concerns, we conduct an extensive evaluation comparing harness evolution with simple test-time scaling and discovery baselines under comparable feedback and inference budgets. We also separate harness search from held-out evaluation to assess generalization. Experiments on Terminal-Bench 2.1 with GPT-5.4 and Claude Opus 4.6 show that automatic harness evolution does not consistently outperform simple test-time scaling methods and exhibits limited generalization when search and evaluation tasks are disjoint. Our results raise important questions about the effectiveness of automatic harness evolution and highlight the need for fairer evaluation protocols and benchmarks for automatic harness design.