Doubly Robust Predictive AI Evaluation

AI model evaluation increasingly relies on benchmark leaderboards, where models are ranked by aggregate scores computed over a response matrix of models against items. In practice, this matrix is almost never fully observed: model developers choose which benchmarks to run, benchmark maintainers choose which models to evaluate, and results are selectively reported. The resulting missingness is systematically non-random rather than incidental, which biases the aggregate scores that leaderboards report. Yet this bias is rarely corrected for when predicting item-level model performance. To reduce this bias, we combine an Item Response Theory (IRT) outcome model with inverse-propensity weighting inside a doubly robust (DR) estimator. A DR estimator retains beneficial theoretical guarantees, such as consistency, as long as either the outcome model or the propensity model is correctly specified. Across nine benchmark settings, the DR estimator drives absolute bias close to zero when missingness is at random, and it remains competitive with, or better than, the naive average and IRT alone when missingness is not at random and across cross-benchmark transfer. The gains are largest where selection is strongest and data are sparse. Our results suggest that DR estimation can improve the quality of reported model performance and yield more reliable comparisons for practitioners deciding which model to use.