No Harness Rules Them All: Harness Choice in Automated Discovery Does Not Generalize

Automated discovery systems such as OpenEvolve and TTT-Discover are often presented as general-purpose search harnesses. In practice, however, these systems bundle many design choices---archives, parent selection, exploration, tree search, and budget allocation---into a single recipe. Because discovery runs are expensive and stochastic, prior evaluations rarely isolate these ingredients with enough independent trials to distinguish robust methodological gains from run-to-run variance. We decompose OpenEvolve-style evolutionary search and TTT-Discover-style PUCT search into simpler building blocks, then evaluate 289 harnesses across 9 model--problem pairs, using over 1.2 million LLM rollouts. Our results show that search harnesses have a generalization problem: no fixed harness is uniformly best across model--problem pairs, and more complex OpenEvolve-style variants often underperform simpler alternatives. Harness choice therefore behaves like a hyperparameter rather than a universally transferable discovery recipe. We also find that early discovery progress predicts final performance, with rank correlations approaching 0.8 by the midpoint of evolution. This motivates EnsembleEvolve, a budget-matched adaptive-allocation experiment. We find that early-score-based pruning improves over these non-adaptive allocation baselines, supporting adaptiveness as a practical response to harness non-transferability.