SCaTR: Simple Calibrated Test-Time Ranking

Test-time scaling (TTS) improves large language models (LLMs) by allocating additional compute at inference time. In practice, TTS is often achieved through parallel scaling: generating multiple candidate responses and selecting the best via a Best-of-$N$ (BoN) strategy. Its effectiveness therefore hinges on the scoring function. Learned scorers such as process reward models (PRMs) can be strong, but they are expensive to train and run. Lightweight confidence heuristics based on token log-probabilities are much cheaper, yet we find that they often perform substantially worse. To improve on lightweight confidence heuristics without incurring the full cost of stronger learned scorers, we introduce SCaTR, a simple and efficient BoN ranking method that learns a lightweight scorer from a small calibration set using hidden representations from the base model. Across coding and mathematical reasoning benchmarks, SCaTR improves over prior confidence-based baselines by up to $\mathbf{9}$%. Relative to LoRA fine-tuning on the same calibration data, it achieves comparable accuracy with up to $\mathbf{8000\times}$ fewer trainable parameters and much lower compute, reducing training and inference latency by up to $\mathbf{150\times}$ and $\mathbf{1000\times}$, respectively. SCaTR is also competitive with strong PRM baselines, and in several settings improves accuracy by up to $\mathbf{7.8}$% on math and $\mathbf{4.2}$% on coding while enabling up to $\mathbf{1000\times}$ faster inference. Overall, SCaTR offers a strong accuracy-efficiency trade-off for scalable test-time selection.