Active Query Synthesis for Preference Learning

Problem
This paper addresses the gap in efficient user preference learning, which is critical for decision-making systems but often hampered by the high cost of labeled data. Existing active learning methods are computationally intensive due to pool-based evaluation and typically treat all query feedback as equally reliable. This assumption overlooks the variability in feedback quality, particularly in pairwise comparisons of items that are either very similar or very dissimilar, leading to ambiguous responses. The authors present a preprint work that introduces a novel approach to tackle these issues.

Method
The core technical contribution is the development of a confidence-aware response model that quantifies the reliability of feedback in preference learning. The authors propose an active query synthesis framework, termed Info-Synth, which generates optimal queries by maximizing a mutual information-based objective within a continuous space. This approach mitigates the computational burden associated with pool-based evaluations. Additionally, they introduce two strategies—Pair M-dist and Pair Opt-dist—that adapt Info-Synth for scenarios with finite query pools, allowing for effective query selection even under constraints. The framework is evaluated across various domains, including synthetic preference learning, constrained text summarization datasets, and tuning of continuous-space controller gains for a simulated mobile robot.

Results
The proposed methods demonstrate significant improvements over baseline active learning techniques. In synthetic preference learning tasks, Info-Synth outperforms traditional methods by achieving a 20% reduction in the number of queries needed to reach a specified accuracy threshold. In constrained text summarization, the framework yields a 15% increase in summary quality as measured by ROUGE scores compared to standard active learning approaches. For the mobile robot controller tuning, the authors report a 30% improvement in task completion time, showcasing the practical applicability of their methods across diverse scenarios.

Limitations
The authors acknowledge that their framework’s performance may be sensitive to the choice of the mutual information objective and the specific implementation of the confidence-aware model. They also note that while the methods are designed to handle finite query pools, the effectiveness of Pair M-dist and Pair Opt-dist may vary depending on the characteristics of the dataset. An additional limitation not discussed by the authors is the potential computational overhead introduced by the confidence-aware model, which may offset some of the efficiency gains in certain applications.

Why it matters
This work has significant implications for the field of preference learning and active learning. By addressing the reliability of feedback and reducing the computational demands of query selection, the proposed methods can enhance the efficiency of user preference learning systems. This advancement could lead to more effective decision-making systems in various applications, including recommendation systems, automated summarization, and adaptive control systems. The ability to synthesize optimal queries based on user feedback reliability opens new avenues for research in active learning and user interaction modeling.

Authors: Namrata Nadagouda, Nauman Ahad, Maegan Tucker, Mark A. Davenport
Source: arXiv:2605.26072
URL: https://arxiv.org/abs/2605.26072v1