Rebellious Student: Reversing Teacher Signals for Reasoning Exploration with Self-Distilled RLVR

Self-distillation has emerged as a powerful framework for post-training LLMs, where a teacher conditioned on privileged information guides a student without it, both from the same model. While this guidance is useful when the student has failed, on successful rollouts, the same mechanism instead overwrites the student's choices and suppresses its own reasoning. We therefore propose reading the self-distillation signal in reverse: when the student succeeds along a path the teacher would not have predicted, these tokens reflect its self-driven reasoning. Building on this, we introduce RLRT (RLVR with Reversed Teacher), which augments GRPO by reinforcing such tokens on correct rollouts. This yields more informative token-level supervision from successful trajectories, enabling more effective use of a fixed rollout and optimization budget. We interpret this as a new form of exploration in RLVR: not uniform diversity, but valuable exploration grounded in the student's own verified success. Across base, instruction-tuned, and thinking-tuned Qwen3 checkpoints, RLRT substantially outperforms self-distillation and exploration-based baselines, establishing information asymmetry as a principled design axis for RLVR.