Learning Long-Horizon LLM Agents from Demonstrations via Test-Time Generative Verifiers

Learning multi-turn LLM agents is challenging, especially in long-horizon environments with sparse and delayed rewards, where meaningful feedback is available only after extended action sequences. As a result, expert demonstrations remain the most practical supervision source, but they are typically used only for supervised fine-tuning (SFT), or behavior cloning. In this work, we ask how demonstrations can be better utilized to improve multi-turn agent performance. Our key idea is to exploit demonstrations in two ways: first, to train a reference policy via SFT, and second, to train a turn-wise generative verifier that judges whether a candidate state-action transition is consistent with expert behavior. At inference time, the verifier guides the frozen SFT policy through turn-level rejection sampling, improving local decisions without additional policy optimization. We further analyze the environment interaction cost of verifier-guided rejection sampling and show that, with an accurate verifier, it is more efficient than Best-of-$N$ for test time scaling in long-horizon settings. Empirically, we achieve great performance on challenging interactive benchmarks and multi-turn tool use cases search-augmented question answering (QA) task only using demonstration data, reaching 96.4\% and 95.5\% success rate on in-domain and out-of-domain tasks (over $+5\%$ over best RL-based method) while reducing interaction cost by up to 4$\times$. On SearchQA, it achieves $+7\%$ over RL-based method and fewer interaction cost compared with the test time scaling. These results demonstrate that properly exploiting demonstration data is a powerful and practical approach for long-horizon agent learning.