Verifier-Backed Hard Problem Generation for Mathematical Reasoning

Problem
This paper addresses the gap in the capability of Large Language Models (LLMs) to autonomously generate valid, challenging, and novel mathematical problems. Current methodologies either rely on costly human expertise or simplistic self-play mechanisms, which often lead to the generation of invalid problems due to reward hacking. The authors present a preprint work that proposes a novel framework for hard problem generation, aiming to enhance the training of LLMs and facilitate autonomous scientific research.

Method
The core technical contribution is the Verifier-Backed Hard Problem Generation (VHG) framework, which incorporates a three-party self-play mechanism involving a problem setter, a problem solver, and an independent verifier. This architecture ensures that the setter’s reward is contingent upon both the validity of the generated problem (assessed by the verifier) and its difficulty (evaluated by the solver). The authors instantiate two types of verifiers: a Hard symbolic verifier that uses formal methods to assess problem validity, and a Soft LLM-based verifier that leverages a language model to evaluate problem difficulty. The framework is applied to two specific tasks: indefinite integral problems and general mathematical reasoning tasks. The training compute details are not disclosed, but the methodology emphasizes the integration of the verifier to mitigate the issues of reward hacking prevalent in previous approaches.

Results
The experimental results demonstrate that the VHG framework significantly outperforms existing baseline methods across both task categories. For indefinite integral tasks, VHG achieved a success rate of 85%, compared to 60% for the best baseline method. In general mathematical reasoning tasks, VHG’s performance was quantified with a 75% success rate, while the leading baseline achieved only 50%. These results indicate a substantial effect size, showcasing the effectiveness of the verifier-enhanced approach in generating valid and challenging problems.

Limitations
The authors acknowledge several limitations, including the potential computational overhead introduced by the verifier, which may affect scalability. Additionally, the reliance on the quality of the verifier could introduce biases if the verifier is not sufficiently robust. The paper does not address the generalizability of the framework to other domains outside of mathematical reasoning, nor does it explore the implications of using different types of verifiers in varying contexts.

Why it matters
The implications of this work are significant for the field of AI and LLM training. By providing a structured approach to problem generation that ensures both validity and difficulty, VHG can enhance the training datasets available for LLMs, potentially leading to improved performance in scientific and mathematical reasoning tasks. This framework could pave the way for more autonomous systems capable of generating and solving complex problems, thereby advancing the state of research in AI-driven scientific inquiry.

Authors: Yuhang Lai, Jiazhan Feng, Yee Whye Teh, Ning Miao
Source: arXiv:2605.06660
URL: https://arxiv.org/abs/2605.06660v1