Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR

Problem
This paper addresses the limitations of traditional rubric-based rewards in reinforcement learning with verifiable rewards (RLVR), particularly in scenarios where multiple qualitative criteria must be satisfied simultaneously. The authors highlight that existing methods often conflate the human-assigned importance of criteria with their effectiveness as optimization signals, leading to suboptimal training outcomes. This work is presented as a preprint and has not yet undergone peer review.

Method
The authors propose POW3R (Policy-Aware Rubric Rewards), a novel framework that adapts the reward weights of rubric criteria during training based on their current relevance to the policy’s performance. POW3R maintains the human-assigned weights and category balance of the rubric while dynamically adjusting criterion-level rewards using rollout-level contrast. This approach emphasizes criteria that effectively differentiate the outputs of the policy, thereby enhancing the informativeness of the GRPO (Generalized Reward with Policy Optimization) reward signal without altering the evaluation target. The framework was evaluated using three base policies across two datasets, which included multimodal and text-only settings.

Results
POW3R demonstrated significant improvements over standard GRPO with static rubric rewards. In a series of 30 comparisons across different base-policy and metric combinations, POW3R achieved victories in 24 instances. Notably, it improved both the mean rubric reward and the strict completion rate (the proportion of prompts meeting all rubric criteria) while requiring 2.5 to 4 times fewer training steps to reach performance plateaus. These results indicate that POW3R effectively enhances the learning process by prioritizing criteria that are currently actionable for the policy.

Limitations
The authors acknowledge that while POW3R improves training efficiency and effectiveness, it may still be sensitive to the initial human-assigned weights and the design of the rubric itself. Additionally, the framework’s reliance on rollout-level contrast could introduce variability based on the quality of the rollouts generated by the policy. The paper does not address potential scalability issues when applied to larger or more complex rubrics, nor does it explore the implications of using POW3R in real-world applications where human feedback may be inconsistent.

Why it matters
The introduction of POW3R has significant implications for the design of reward systems in reinforcement learning, particularly in applications requiring nuanced qualitative assessments. By decoupling the importance of rubric criteria from their current utility as optimization signals, this work paves the way for more effective training methodologies that can adapt to the evolving capabilities of the policy. This adaptability could lead to more robust and efficient learning in complex environments, ultimately enhancing the performance of AI systems in real-world tasks.

Authors: Utkarsh Tyagi, Xingang Guo, MohammadHossein Rezaei, Daniel George, Anas Mahmoud, Jackson Lee, Bing Liu, Yunzhong He
Source: arXiv:2605.20164
URL: https://arxiv.org/abs/2605.20164v1