Causal Explanations from the Geometric Properties of ReLU Neural Networks

Problem
This paper addresses the gap in interpretability of neural networks, particularly in the context of reinforcement learning (RL) for autonomous systems. The authors highlight that existing methods for generating causal explanations often rely on distilled models, which compromise performance and do not accurately reflect the decision-making processes of the original networks. This lack of fidelity in explanations poses challenges for safety assurance in autonomous systems. The work is presented as a preprint and has not yet undergone peer review.

Method
The authors propose a novel approach to generating causal explanations by leveraging the geometric properties of ReLU (Rectified Linear Unit) neural networks. They utilize the fact that a ReLU network can be represented as a piecewise linear function, partitioning the input space into regions defined by an n-dimensional convex polytope. The core technical contribution involves extracting causal rules directly from this geometric representation, allowing for explanations that are inherently tied to the original model’s behavior. The method does not require distillation, thus preserving the performance characteristics of the original network. The paper does not disclose specific details regarding the architecture, loss functions, or training compute used in their experiments.

Results
The authors demonstrate the effectiveness of their approach through empirical evaluations, although specific numerical results and comparisons to named baselines are not provided in the abstract. They claim that their method yields causal explanations that are more accurate and reliable than those derived from distilled models. The implications of their findings suggest a significant improvement in the interpretability of neural networks without sacrificing performance, although exact effect sizes and benchmark comparisons are not detailed in the provided text.

Limitations
The authors acknowledge that their method is contingent on the geometric properties of ReLU networks, which may not generalize to other activation functions or architectures. Additionally, they do not address potential scalability issues when applying their approach to larger networks or more complex tasks. The lack of quantitative results in the abstract limits the ability to assess the practical impact of their method compared to existing techniques.

Why it matters
This work has significant implications for the fields of eXplainable Artificial Intelligence (XAI) and eXplainable Reinforcement Learning (XRL). By providing a method for generating causal explanations that are directly tied to the original model’s geometry, the authors contribute to the development of safer and more interpretable autonomous systems. This could enhance trust in AI systems, particularly in safety-critical applications, and pave the way for further research into geometric interpretations of neural networks and their applications in explainability.

Authors: Hector Woods, Philippa Ryan, Rob Alexander
Source: arXiv:2605.10396
URL: https://arxiv.org/abs/2605.10396v1