RUBEN: Rule-Based Explanations for Retrieval-Augmented LLM Systems

Problem
This paper addresses the lack of interpretable explanations for the outputs of retrieval-augmented large language models (LLMs), particularly in data-driven applications. The authors identify a gap in existing literature regarding the ability to generate minimal, rule-based explanations that can effectively elucidate model behavior. The work is presented as a preprint and has not yet undergone peer review.

Method
The core technical contribution of this paper is the development of RUBEN, an interactive tool that employs novel pruning strategies to derive a minimal set of rules that can explain the outputs of retrieval-augmented LLMs. The authors utilize a rule-based approach to identify and subsume redundant rules, thereby enhancing interpretability. The methodology includes a systematic exploration of rule generation and pruning techniques, although specific architectural details of the LLMs used for evaluation are not disclosed. The authors also introduce applications of these rules in assessing LLM safety, particularly focusing on the resiliency of safety training and the effectiveness of adversarial prompt injections. The training compute required for the rule extraction process is not specified.

Results
RUBEN demonstrates significant improvements in interpretability over baseline methods, achieving a reduction in the number of rules needed to explain model outputs by up to 70% compared to traditional rule extraction techniques. The authors benchmark their approach against existing rule-based explanation methods, although specific baseline models are not named. The effectiveness of RUBEN is further validated through its application in safety testing, where it successfully identifies vulnerabilities in LLMs against adversarial prompts, showcasing a marked increase in detection rates of unsafe outputs.

Limitations
The authors acknowledge several limitations, including the potential for overfitting in rule generation, which may lead to explanations that do not generalize well across different contexts. Additionally, the reliance on specific retrieval-augmented LLM architectures may limit the applicability of RUBEN to other model types. The paper does not address the computational overhead associated with the rule extraction process, which could be a concern in real-time applications. Furthermore, the evaluation of safety applications is preliminary and may require more extensive testing across diverse datasets and adversarial scenarios.

Why it matters
The implications of this work are significant for the field of explainable AI, particularly in enhancing the transparency of LLMs in critical applications such as healthcare, finance, and autonomous systems. By providing a systematic approach to rule-based explanations, RUBEN could facilitate better understanding and trust in LLM outputs, thereby improving user confidence and compliance with regulatory standards. The ability to test LLM safety through interpretable rules also opens avenues for future research in adversarial robustness and model safety, potentially leading to more resilient AI systems.

Authors: Joel Rorseth, Parke Godfrey, Lukasz Golab, Divesh Srivastava, Jarek Szlichta
Source: arXiv:2605.10862
URL: https://arxiv.org/abs/2605.10862v1