Towards Full Pipeline FP8 Reinforcement Learning for LLMs

Reinforcement learning (RL) is crucial for enhancing the reasoning and agentic capabilities of large language models (LLMs). FP8 quantization offers a promising solution to accelerate RL training, but achieving stable FP8 RL remains challenging. While previous works have focused on resolving train–inference mismatches using correction techniques like TIS, we reveal that full-pipeline FP8 RL still suffers from severe training instability, manifesting as anomalous mid-training entropy surges and garbled outputs. We trace this instability to a previously overlooked cause: compounded FP8 quantization noise distorts the importance ratio, disproportionately pushing negative-advantage tokens outside the trust region and erroneously zeroing out their gradients. As a result, pathological outputs are not properly penalized and accumulate over the course of training. To address this, we propose Calibrated Clipping, a dynamic method that aligns the FP8 clipping bounds with high-precision BF16 distributions by matching the lower-bound clipping quantile and re-balancing the upper bound accordingly. Extensive experiments across GRPO and DAPO algorithms, model scales from 8B to 32B, and multiple FP8 scaling granularities demonstrate that our approach successfully eliminates entropy surges and restores performance comparable to the BF16 baseline, while achieving up to 1.5× throughput improvement in the training phase.