Reproducing Complex Set-Compositional Information Retrieval

Problem
This paper addresses the gap in understanding how current information retrieval (IR) paradigms handle complex set-compositional queries, which involve conjunction, disjunction, and exclusion. The authors highlight that existing retrieval methods may exploit “semantic shortcuts” rather than genuinely satisfying the constraints of such queries. This work is presented as a reproducibility study, benchmarking major retrieval families and reasoning-targeted methods on the QUEST dataset and its variants, while introducing a new benchmark called LIMIT+.

Method
The authors benchmark various retrieval methods, including neural retrievers and reasoning-targeted approaches, on the QUEST and LIMIT+ datasets. LIMIT+ is designed to evaluate retrieval effectiveness based on arbitrary attribute predicates and constraint satisfaction, minimizing reliance on pretrained knowledge. The study employs standard metrics such as Recall@100 to assess performance. The authors also stratify results by compositional depth to analyze how different methods perform as query complexity increases. The paper includes the release of code and data generation scripts to facilitate reproducibility and controlled evaluation.

Results
On the QUEST benchmark, the best-performing neural retrievers achieve a Recall@100 of over 0.41, significantly outperforming BM25, which achieves a Recall@100 of 0.20. However, reasoning-targeted methods like ReasonIR and Search-R1 do not consistently outperform general-purpose retrievers. In contrast, on the LIMIT+ benchmark, the strongest method from QUEST experiences a drastic drop in performance, with Recall@100 collapsing from approximately 0.42 to below 0.02, while classic lexical retrieval methods improve to around 0.96. The analysis of compositional depth reveals a consistent degradation in performance across all methods, with algebraic sparse and lexical methods demonstrating more stable performance compared to dense approaches, which exhibit significant collapse.

Limitations
The authors acknowledge that their findings are limited to the specific datasets used (QUEST and LIMIT+) and may not generalize to other retrieval contexts. They also note that the performance of reasoning-targeted methods is inconsistent, suggesting that these methods may not be universally applicable. Additionally, the study does not explore the impact of different training regimes or hyperparameter settings on the performance of the evaluated methods, which could provide further insights into their capabilities.

Why it matters
This work has significant implications for the development of future IR systems, particularly in contexts requiring complex query handling. The introduction of the LIMIT+ benchmark provides a controlled environment for evaluating retrieval methods against compositional constraints, which could lead to more robust and effective retrieval systems. The findings challenge the effectiveness of reasoning-targeted methods, suggesting that general-purpose retrievers may be more suitable for certain tasks. This research encourages further exploration into the limitations of current paradigms and the need for improved methodologies in handling complex information needs.

Authors: Vincent Degenhart, Dewi Timman, Arjen P. de Vries, Faegheh Hasibi, Mohanna Hoveyda
Source: arXiv:2605.03824
URL: https://arxiv.org/abs/2605.03824v1