Think Less, Solve More: Efficient Reasoning via Clipped Process Rewards

Large reasoning models trained with reinforcement learning from verifiable outcome-level rewards (RLVR) are known to exhibit overthinking, generating longer chains of thought even when the correct answer has been found. Existing techniques mainly treat this as a length control problem, but such length-focused approaches do not address the underlying cause and can be gamed in ways that reduce length without improving reasoning quality. Process reward models (PRMs) offer denser step-level feedback than a single terminal correctness signal, but integrating PRM supervision into critic-free, GRPO-style training raises the question of where within the trajectory the process signal should be estimated, and in what functional form. In this study, we restrict process reward estimation to the post-think segment of the trajectory, and empirically study the effect of \textit{clipping} the raw PRM score into a penalty-only signal and \textit{scaffolding} the process supervision to the first or last few steps of the post-think segment. We show that a mean aggregation over a clipped process reward, concentrated toward the end of the post-think segment, improves both accuracy and token efficiency in GRPO. For example, DeepSeek-R1-Distill-Qwen-1.5B trained with our method shows a 14-point gain in pass@32 on AIME24 while reducing the token usage by 7\%.