Beyond Repeated Sampling: Learning Search Policies for LLM Reasoning

Problem

This work addresses the limitations of naive repeated sampling in large language models (LLMs) for reasoning tasks. The authors highlight that traditional methods often fail to effectively leverage the reasoning capabilities of LLMs, leading to suboptimal performance in generating accurate answers. This paper is a preprint and has not undergone peer review.

Method

The proposed method consists of two main components: Concept Generation and Answer Generation.

• Concept Generation: This component is optimized using reinforcement learning techniques to maximize the downstream success of answer generation. The concept generator is trained to produce diverse concepts within a single trajectory, enhancing the variety and relevance of the generated concepts.
• Answer Generation: A larger, frozen model is employed, which is conditioned on the concepts generated by the concept generator. This architecture allows the answer generator to utilize a broader context provided by the diverse concepts, improving the quality of the responses.

Results

The results indicate a substantial improvement in performance metrics, specifically in the pass@k metric, when compared to the baseline of naive repeated sampling. Additionally, the concept generator demonstrates effective transferability, performing well with answer generators that it was not explicitly trained against, including those from different model families. However, the available text does not report quantitative results.

Limitations

The authors do not report any limitations in their work. However, it is important to note that the lack of quantitative results may hinder a comprehensive evaluation of the method's effectiveness across various scenarios.

Why it matters

This research has significant implications for downstream work in the field of natural language processing and reasoning. By optimizing the concept generation process through reinforcement learning, the proposed method enhances the reasoning capabilities of LLMs, potentially leading to more accurate and contextually relevant outputs. This advancement could pave the way for improved applications in areas such as automated reasoning, question answering, and interactive AI systems.