Beyond GRPO and On-Policy Distillation: An Empirical Sparse-to-Dense Reward Principle for Language-Model Post-Training

Problem
This preprint addresses the inefficiencies in the allocation of labeled verifiable training data in reinforcement learning (RL) settings, particularly in the context of language model post-training. The authors critique the conventional approach of applying sparse sequence-level rewards directly to deployment models, such as through Generalized Reward Policy Optimization (GRPO). They propose that this method neglects a reward-density principle, which suggests that different reward regimes (sparse vs. dense) should be strategically employed based on the model’s training phase and objectives.

Method
The core technical contribution is the introduction of an empirical sparse-to-dense reward principle that optimally allocates labeled data across different training stages. The authors propose a two-step training process: first, using sparse rewards for teacher models (e.g., Qwen3 and Llama) to facilitate exploration and behavior discovery; second, transferring this learned behavior to a smaller student model through dense supervision. The authors implement a forward-KL warmup on teacher rollouts, followed by On-Policy Distillation (OPD) on student rollouts, which serves as a “bridge” to enhance the effectiveness of subsequent student-side sparse RL. The experiments utilize a fixed Qwen3-1.7B deployment student and an 8B teacher model, with specific attention to the MATH benchmark for evaluation.

Results
The results demonstrate that the proposed method significantly outperforms traditional GRPO approaches. Specifically, the RL-improved 8B teacher distilled through the dense bridge achieves a performance of 78.5% on the MATH benchmark, compared to 75.4% with GRPO on a cold student. This represents a 3.1 percentage point improvement. Additionally, the authors report that the bridge mechanism enhances the effectiveness of student-side sparse RL, yielding a 2.8 point advantage over a matched replay control. The findings indicate that the bridge is crucial for maximizing the utility of labeled data and improving model performance.

Limitations
The authors acknowledge that their approach may not generalize across all types of tasks or models, as the empirical results are primarily focused on specific language models and the MATH benchmark. They do not address potential scalability issues when applying this method to larger or more complex models. Furthermore, the reliance on a two-step training process may introduce additional computational overhead, which could be a concern in resource-constrained environments.

Why it matters
This work has significant implications for the field of RL and language model training, particularly in scenarios where labeled data is scarce. By establishing a clear framework for the strategic allocation of rewards, the authors provide a novel perspective that could enhance the efficiency of model training and improve performance on downstream tasks. The proposed sparse-to-dense reward principle encourages further exploration of hybrid training methodologies, potentially leading to more robust and efficient language models in various applications.

Authors: Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, Zhipeng Wang, Alborz Geramifard
Source: arXiv:2605.12483
URL: https://arxiv.org/abs/2605.12483v1