Rebellious Student: Reversing Teacher Signals for Reasoning Exploration with Self-Distilled RLVR

Problem
This paper addresses the limitations of traditional self-distillation in reinforcement learning (RL) for large language models (LLMs), particularly in scenarios where the teacher model’s guidance can suppress the student’s independent reasoning. The authors identify a gap in the literature regarding the effective utilization of successful student rollouts that diverge from teacher predictions. This work is presented as a preprint and has not yet undergone peer review.

Method
The authors propose a novel approach called RLRT (Reinforcement Learning with Reversed Teacher), which modifies the self-distillation framework by reversing the teacher signals. Instead of merely guiding the student based on the teacher’s predictions, RLRT reinforces the tokens generated by the student during successful rollouts that the teacher would not have anticipated. This method builds on the Generalized Reinforcement Policy Optimization (GRPO) framework, enhancing it by focusing on the student’s self-driven reasoning. The training process involves augmenting the reward structure to prioritize these successful, unexpected tokens, thereby promoting exploration that is not uniform but rather informed by the student’s own achievements. The paper does not disclose specific training compute details.

Results
The proposed RLRT method demonstrates significant performance improvements over traditional self-distillation and exploration-based baselines across various Qwen3 checkpoints, including base, instruction-tuned, and thinking-tuned models. The authors report substantial effect sizes, although specific numerical results and comparisons to named baselines are not detailed in the abstract. The improvements suggest that leveraging information asymmetry—where the student’s successful reasoning diverges from the teacher’s predictions—can lead to more effective exploration strategies in RLVR.

Limitations
The authors acknowledge that their approach may not generalize across all types of tasks or models, as it relies on the assumption that successful rollouts can be identified and effectively reinforced. They do not address potential issues related to the scalability of the method or the computational overhead introduced by the reversed signal processing. Additionally, the reliance on the teacher’s predictions may still impose limitations on the diversity of exploration, as it is contingent on the teacher’s initial training and capabilities.

Why it matters
This work has significant implications for the design of RL systems, particularly in the context of LLMs. By establishing information asymmetry as a new design axis for RLVR, the authors open avenues for future research that could explore more nuanced forms of exploration and exploitation in RL settings. The ability to harness the student’s independent reasoning could lead to more robust and adaptable models, enhancing their performance in complex tasks where traditional guidance may fall short. This approach could also inspire new methodologies in self-supervised learning and other areas where teacher-student dynamics are prevalent.

Authors: Jeonghye Kim, Jiwon Jeon, Dongsheng Li, Yuqing Yang
Source: arXiv:2605.10781
URL: https://arxiv.org/abs/2605.10781v1