Boosting Reinforcement Learning with Verifiable Rewards via Randomly Selected Few-Shot Guidance

Problem
This paper addresses the limitations of Reinforcement Learning with Verifiable Rewards (RLVR) in terms of sample efficiency, particularly in complex tasks where generating correct rollouts is challenging. While RLVR has shown promise in training Large Language Models (LLMs) for tasks like math and coding, existing methods often rely on extensive Supervised Fine-Tuning (SFT) when RL fails, which can be data-intensive and costly. The authors propose a novel approach, FEST, which leverages a few-shot demonstration strategy to mitigate the data requirements of SFT. This work is presented as a preprint and has not yet undergone peer review.

Method
FEST (FEw-ShoT demonstration-guided RLVR) introduces a framework that utilizes only 128 randomly selected demonstrations from a larger SFT dataset. The core technical contributions include:

1. Supervised Signal: Incorporating a supervised learning component to guide the RL process.
2. On-Policy Signal: Utilizing on-policy data to enhance the learning process during RL.
3. Decaying Weights: Implementing a decay mechanism on the few-shot SFT dataset to prevent overfitting during multiple epochs of training. The authors do not disclose specific architectural details or training compute but emphasize the efficiency of their approach in leveraging minimal data.

Results
FEST demonstrates significant improvements over baseline methods across several benchmarks, achieving comparable performance to those using full SFT datasets while utilizing substantially less data. The authors report effect sizes that indicate FEST outperforms traditional SFT methods by a notable margin, although specific numerical results and comparisons to named baselines are not detailed in the abstract. The results suggest that FEST can effectively bridge the gap between RL and SFT, enhancing sample efficiency in RLVR applications.

Limitations
The authors acknowledge that the reliance on a limited number of demonstrations may not generalize well to all tasks, particularly those requiring diverse or complex reasoning. Additionally, the decay mechanism’s effectiveness may vary depending on the specific task and dataset characteristics. The paper does not address potential scalability issues when applied to larger datasets or more complex models, nor does it explore the impact of varying the number of demonstrations beyond the tested 128.

Why it matters
The implications of this work are significant for the field of reinforcement learning, particularly in applications involving LLMs. By demonstrating that effective RL can be achieved with minimal data through few-shot learning, this research opens avenues for more efficient training methodologies that reduce the dependency on large labeled datasets. This could lead to broader accessibility of RL techniques in various domains, including natural language processing and beyond, where data acquisition is often a bottleneck. The findings encourage further exploration of hybrid approaches that combine RL and SFT, potentially leading to more robust and sample-efficient learning paradigms.

Authors: Kai Yan, Alexander G. Schwing, Yu-Xiong Wang
Source: arXiv:2605.15012
URL: https://arxiv.org/abs/2605.15012v1