BalanceRAG: Joint Risk Calibration for Cascaded Retrieval-Augmented Generation

Problem
This paper addresses the gap in the literature regarding the calibration of cascaded retrieval-augmented generation (RAG) systems, particularly in the context of large language models (LLMs). While RAG can enhance factuality, applying it indiscriminately to all queries is inefficient, especially when LLM-only responses are reliable. The authors propose a method to jointly calibrate uncertainty thresholds for LLM-only and RAG branches, which is crucial for optimizing the trade-off between response accuracy and retrieval overhead. This work is presented as a preprint and has not yet undergone peer review.

Method
The core technical contribution of this paper is the development of BalanceRAG, which implements a joint risk calibration framework for cascaded RAG systems. The method involves two main components: (1) uncertainty scoring from both the LLM-only and RAG branches, and (2) a two-dimensional lattice representation of threshold pairs. BalanceRAG employs sequential graphical testing to identify safe operating points on this lattice, allowing for risk-adaptive threshold calibration. This approach ensures that the system maintains a target error rate while maximizing the number of accepted queries. Additionally, BalanceRAG introduces multi-risk calibration, which allows for the simultaneous management of retrieval usage and selection-conditioned risk. The authors do not disclose specific training compute or data details.

Results
BalanceRAG was evaluated on three open-domain question answering benchmarks, demonstrating significant improvements over baseline methods. The results indicate that BalanceRAG effectively meets prescribed risk levels while achieving higher coverage and accepting more correct responses. Specifically, it reduces unnecessary retrieval calls compared to a baseline that employs RAG for all queries. The paper reports that BalanceRAG outperforms traditional RAG systems in terms of both accuracy and efficiency, although exact numerical results and comparisons to specific baselines are not detailed in the summary.

Limitations
The authors acknowledge that the calibration process may still be conservative, potentially leading to missed opportunities for retrieval in cases where the LLM-only branch could have provided a reliable answer. They also note that the performance of BalanceRAG is contingent on the quality of uncertainty scoring from both branches. An obvious limitation not discussed by the authors is the potential computational overhead introduced by the calibration process itself, which may offset some efficiency gains in specific scenarios.

Why it matters
The implications of this work are significant for the design of efficient retrieval-augmented systems. By enabling joint risk calibration, BalanceRAG allows for more intelligent decision-making in query handling, potentially leading to reduced latency and resource consumption in real-world applications. This approach could inform future research on adaptive retrieval strategies and improve the deployment of LLMs in production environments, where balancing accuracy and efficiency is critical.

Authors: Zijun Jia, Yuanchang Ye, Sen Jia, Yiyao Qian, Haoning Wang, Baojie Chen, Diyin Tang, Jinsong Yu et al.
Source: arXiv:2605.20084
URL: https://arxiv.org/abs/2605.20084v1